You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "sarfarazahammed (via GitHub)" <gi...@apache.org> on 2023/08/08 07:20:24 UTC
[GitHub] [beam] sarfarazahammed opened a new issue, #27892: [Bug]: ignoreUnknownValues not working when using withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED).withSchema(successVisitorTableSchema)
sarfarazahammed opened a new issue, #27892:
URL: https://github.com/apache/beam/issues/27892
### What happened?
We are currently executing a dataflow operation to transfer data from Kafka to BigQuery. Within this data flow, we have established a predetermined schema for BigQuery, which we use for generating new tables. The event from Kafka is updated with a new schema, our intention is to maintain the data in the existing tables while disregarding the newly introduced values.
In our initial implementation using **STREAMING_INSERTS**, we effectively addressed this scenario by configuring **ignoreUnknownValues** as true. This approach allowed us to manage this use case. However, upon transitioning to **STORAGE_WRITE_API** to achieve precisely once inserts, we encountered a challenge. Specifically, the aforementioned approach of utilizing ignoreUnknownValues is no longer effective, resulting in potential data loss.
The error we are encountering is as follows:
```
com.google.cloud.bigquery.storage.v1.ConnectionWorker doneCallback
Connection finished with error com.google.api.gax.rpc.InvalidArgumentException:
io.grpc.StatusRuntimeException:
INVALID_ARGUMENT:
Input schema has more fields than BigQuery schema, extra fields: 'testfield' Entity: projects/{projectId}/{dataset}/tables/{tableName}/streams/{streamId} for stream projects/{projectId}/datasets/{staging_events_visitor}/tables/{tableId}/streams/{streamId} with write id: {writeId}
```
Steps to produce it:
1. Create a `TableSchema`
2. use ```.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED).withSchema(schema).withMethod(BigQueryIO.Write.Method.STORAGE_WRITE_API)``` while writing to BQ
4. Execute beam to transfer some data with the current schema
5. Add new field to `TableSchema`
6. Try to execute pipeline that consumes event with new field in it and adds it to existing table.
7. You will see the error
### Issue Priority
Priority: 1 (data loss / total loss of function)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] sarfarazahammed commented on issue #27892: [Bug]: ignoreUnknownValues not working when using withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED).withSchema(successVisitorTableSchema)
Posted by "sarfarazahammed (via GitHub)" <gi...@apache.org>.
sarfarazahammed commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-1669051506
.add-labels bigquery, io
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on issue #27892: [Bug]: ignoreUnknownValues not working when using withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED).withSchema(successVisitorTableSchema)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-1669052344
Label cannot be managed because it does not exist in the repo. Please check your spelling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED [beam]
Posted by "vamsiprasanth-narvar (via GitHub)" <gi...@apache.org>.
vamsiprasanth-narvar commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-2746851320
Is the above going to be fixed or it would be open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] sarfarazahammed commented on issue #27892: [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED
Posted by "sarfarazahammed (via GitHub)" <gi...@apache.org>.
sarfarazahammed commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-1670526737
.add-labels bigquery
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #27892: [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-1716779016
> Is the problem that the table is precreated but the schema passed into BigQueryIO contains fields not in the actual table?
That's what this reads like to me, yes. So I guess it would be a different option, if we wanted to add it. Specifically it could allow someone's cronjob to keep functioning though dropping data if they update the schema of the table it targets?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-5455167956
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] sarfarazahammed commented on issue #27892: [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED
Posted by "sarfarazahammed (via GitHub)" <gi...@apache.org>.
sarfarazahammed commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-1716900429
> So this is an expected use of the sink. It wasn't really an expected us of STREAMING_INSERTS either, however since STREAMING_INSERTS does absolutely no schema vaidation, it just happened to work.
I agree with you on this but but it seems like the way things are currently set up, the system is double-checking the schema before actually writing data. So, if we have a field in the BQ table and the json we're sending has data for that field, but the schema provided in the request doesn't include details about that field, it's storing the value as null. I'm curious, is this the intended behavior? Is there a reason behind it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-5870246072
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on issue #27892: [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-1718714198
The storage write API requires us to pass in a schema when we open a connection to it, and that schema must match the actual table schema. This is the schema you pass into withSchema. ignoreUnknownValues allows input element to have fields that don't match the schema, in which case Beam will simply ignore those extra fields. This scenario worked with STREAMING_INSERTS because the old STREAMING_INSERTS api didn't require us to pass in a schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on issue #27892: [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-1716411894
Trying to understand the actual issue. Is the problem that the table is precreated but the schema passed into BigQueryIO contains fields not in the actual table? ignoreUnknownValues is meant for the case when the "records" contain unknown values - the assumption was that the value passed into withSchema matched the actual table schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] sarfarazahammed commented on issue #27892: [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED
Posted by "sarfarazahammed (via GitHub)" <gi...@apache.org>.
sarfarazahammed commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-1670527099
.add-labels io
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #27892: [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-1715744263
@abacn @reuvenlax what are your thoughts on this change in options for the new API?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on issue #27892: [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on issue #27892:
URL: https://github.com/apache/beam/issues/27892#issuecomment-1716866954
So this is an expected use of the sink. It wasn't really an expected us of STREAMING_INSERTS either, however since STREAMING_INSERTS does absolutely no schema vaidation, it just happened to work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #27892: [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED
URL: https://github.com/apache/beam/issues/27892
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org