You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "prodriguezdefino (via GitHub)" <gi...@apache.org> on 2024/09/20 19:44:38 UTC
[PR] [WIP] Enable CDC configuration for cross language StorageWrite API usage [beam]
prodriguezdefino opened a new pull request, #32523:
URL: https://github.com/apache/beam/pull/32523
This change enables the configuration of CDC usage for the `BigQueryStorageWriteApiSchemaTransformProvider`.
By setting `configuration.setUseCDCWrites(true)` on the provider's config, the transform creation will check for a Row schema like:
```
{
cdc_info:{
mutation_type: "..." // can be UPSERT or DELETE
change_sequence_number: "..." // used for custom insert ordering
}
record:{...} // actual data to be inserted
}
```
The implementation also enables the possibility of setting a dynamic destination alongside with CDC usage by using this Row schema:
```
{
cdc_info:{
mutation_type: "..." // can be UPSERT or DELETE
change_sequence_number: "..." // used for custom insert ordering
}
destination: "..." // destination table for the data
record:{...} // actual data to be inserted
}
```
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Enable CDC configuration for cross language StorageWrite API usage [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #32523:
URL: https://github.com/apache/beam/pull/32523#issuecomment-2364726505
fixes #32527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Enable CDC configuration for cross language StorageWrite API usage [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #32523:
URL: https://github.com/apache/beam/pull/32523#issuecomment-2366929168
completed python integration on #32529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Enable CDC configuration for cross language StorageWrite API usage [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino closed pull request #32523: [WIP] Enable CDC configuration for cross language StorageWrite API usage
URL: https://github.com/apache/beam/pull/32523
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Enable CDC configuration for cross language StorageWrite API usage [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #32523:
URL: https://github.com/apache/beam/pull/32523#issuecomment-2365037662
@ahmedabu98 FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org