You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ragyibrahim (via GitHub)" <gi...@apache.org> on 2023/02/07 04:33:23 UTC

[GitHub] [beam] ragyibrahim opened a new issue, #25356: [Feature Request]: Mechanism to enable BigQuery IO to handle table schema changes while a streaming job is running

ragyibrahim opened a new issue, #25356:
URL: https://github.com/apache/beam/issues/25356

   ### What would you like to happen?
   
   Currently, the BQ IO can not handle schema changes if a table schema changes during a streaming job the IO returns an error and the rows are dropped. 
   We propose a new mechanism that will enable BQ to handle a schema change:
   
   1. On startup a check will be made to ensure table schemas are up to date, if not, update the current schema
   2. Upon receiving a schema error, check if there are any columns in the schema supplied to the BQ IO step that is not in the existing table schema, the column(s) will be added to the table
   
   @ tomlynchRNA
   @ ahmedabu98
   
   ### Issue Priority
   
   Priority: 2 (default / most feature requests should be filed as P2)
   
   ### Issue Components
   
   - [X] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [X] Component: IO connector
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ragyibrahim commented on issue #25356: [Feature Request]: Mechanism to enable BigQuery IO to handle table schema changes while a streaming job is running

Posted by "ragyibrahim (via GitHub)" <gi...@apache.org>.
ragyibrahim commented on issue #25356:
URL: https://github.com/apache/beam/issues/25356#issuecomment-1420188302

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org