You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "rohitsinha54 (via GitHub)" <gi...@apache.org> on 2024/07/08 22:19:18 UTC
[PR] Introduce support for emitting lineage in BQ Source [beam]
rohitsinha54 opened a new pull request, #31805:
URL: https://github.com/apache/beam/pull/31805
- Adds a Helper class which wraps around StringSet metrics to define sources/sinks metrics for lineage.
- Add support in BQ IO Java SDK sources to emit the dataset they are accessing as lineage via stringset metrics.
addresses https://github.com/apache/beam/issues/31804
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce support for emitting lineage in BQ Source [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #31805:
URL: https://github.com/apache/beam/pull/31805#discussion_r1669442776
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java:
##########
@@ -106,15 +109,21 @@ public List<BigQueryStorageStreamSource<T>> split(
@Nullable Table targetTable = getTargetTable(bqOptions);
Review Comment:
I *think* step-level counters are available in split, but we should verify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce support for emitting lineage in BQ Source [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb merged PR #31805:
URL: https://github.com/apache/beam/pull/31805
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce support for emitting lineage in BQ Source [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31805:
URL: https://github.com/apache/beam/pull/31805#issuecomment-2215448785
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce support for emitting lineage in BQ Source [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #31805:
URL: https://github.com/apache/beam/pull/31805#issuecomment-2220648756
It causes Java PostCommit to fail due to same reason of #31814 . When there is StringSet counter in the pipeline, Dataflow legacy runner pipeline fails updating work item. This is due to backend not supporting it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce support for emitting lineage in BQ Source [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #31805:
URL: https://github.com/apache/beam/pull/31805#discussion_r1688811790
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java:
##########
@@ -152,9 +155,10 @@ public List<BoundedSource<T>> split(long desiredBundleSizeBytes, PipelineOptions
if (cachedSplitResult == null) {
ExtractResult res = extractFiles(options);
LOG.info("Extract job produced {} files", res.extractedFiles.size());
-
if (res.extractedFiles.size() > 0) {
BigQueryOptions bqOptions = options.as(BigQueryOptions.class);
+ // emit this table ID as a lineage source
+ Lineage.getSources().add(BigQueryHelpers.dataCatalogName(getTableToExtract(bqOptions)));
Review Comment:
BigQueryQuerySource.getTableToExtract will call BigQueryQuerySource.getTableReference run a BigQuery query job, which is expensive. This caused internal test error
We should cache the result of `getTableToExtract`, or get this name from elsewhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce support for emitting lineage in BQ Source [beam]
Posted by "rohitsinha54 (via GitHub)" <gi...@apache.org>.
rohitsinha54 commented on PR #31805:
URL: https://github.com/apache/beam/pull/31805#issuecomment-2215447479
R: @robertwb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org