You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/14 05:58:01 UTC
[GitHub] [beam] youngoli opened a new pull request, #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
youngoli opened a new pull request, #21852:
URL: https://github.com/apache/beam/pull/21852
The expansion service now reads the Project pipeline option so that during expansion it can perform GCP operations on the correct project. Also un-sickbay a test that was blocked on this issue.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1158347359
Run XVR_GoUsingJava_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1154750727
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1154749507
Run XVR_GoUsingJava_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1173198559
Run XVR_GoUsingJava_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1211315529
Looks like it still fails to get the projectID, but it could be just in how the Go SDK is providing it in pipeline options (or not).
`14:35:37 Caused by: java.lang.NullPointerException: Required parameter projectId must be specified.`
```
14:35:37 --- FAIL: TestBigQueryIO_BasicWriteQueryRead (313.80s)
14:35:37 panic: tried cross-language for beam:transform:org.apache.beam:schemaio_bigquery_read:v1 against localhost:37161 and failed
14:35:37 expanding external transform
14:35:37 expanding transform with ExpansionRequest: components:{environments:{key:"go" value:{}}} transform:{unique_name:"External" spec:{urn:"beam:transform:org.apache.beam:schemaio_bigquery_read:v1" payload:"\nX\n\x0e\n\x08location\x1a\x02\x10\x07\n\x0c\n\x06config\x1a\x02\x10\t\n\x12\n\ndataSchema\x1a\x04\x08\x01\x10\t\x12$3fc24beb-ef0b-4fd0-b491-64340b33ca1e\x12k\x03\x01\x04\x00f\x04\x01\rbSELECT * FROM `apache-beam-testing.beam_bigquery_io_test_temp.go_bqio_it_temp_1660164801805738010`"} environment_id:"go"} namespace:"VkgVLcucsq"
14:35:37 expansion failed
14:35:37 caused by:
14:35:37 org.apache.beam.sdk.io.gcp.bigquery.BigQuerySchemaRetrievalException: Exception while trying to retrieve schema of query
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySourceDef.getBeamSchema(BigQueryQuerySourceDef.java:179)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead.expand(BigQueryIO.java:1226)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead.expand(BigQueryIO.java:747)
14:35:37 at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548)
14:35:37 at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:482)
14:35:37 at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySchemaIOProvider$BigQuerySchemaIO$1.expand(BigQuerySchemaIOProvider.java:183)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySchemaIOProvider$BigQuerySchemaIO$1.expand(BigQuerySchemaIOProvider.java:165)
14:35:37 at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548)
14:35:37 at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:499)
14:35:37 at org.apache.beam.sdk.expansion.service.ExpansionService$TransformProvider.apply(ExpansionService.java:396)
14:35:37 at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:516)
14:35:37 at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:603)
14:35:37 at org.apache.beam.model.expansion.v1.ExpansionServiceGrpc$MethodHandlers.invoke(ExpansionServiceGrpc.java:220)
14:35:37 at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
14:35:37 at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:354)
14:35:37 at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:866)
14:35:37 at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
14:35:37 at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
14:35:37 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
14:35:37 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
14:35:37 at java.lang.Thread.run(Thread.java:750)
14:35:37 Caused by: java.lang.NullPointerException: Required parameter projectId must be specified.
14:35:37 at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:907)
14:35:37 at com.google.api.client.util.Preconditions.checkNotNull(Preconditions.java:138)
14:35:37 at com.google.api.services.bigquery.Bigquery$Jobs$Insert.<init>(Bigquery.java:1737)
14:35:37 at com.google.api.services.bigquery.Bigquery$Jobs.insert(Bigquery.java:1687)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.dryRunQuery(BigQueryServicesImpl.java:473)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryQueryHelper.dryRunQueryIfNeeded(BigQueryQueryHelper.java:73)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySourceDef.getBeamSchema(BigQueryQuerySourceDef.java:168)
14:35:37 ... 21 more
14:35:37 [recovered]
14:35:37 panic: tried cross-language for beam:transform:org.apache.beam:schemaio_bigquery_read:v1 against localhost:37161 and failed
14:35:37 expanding external transform
14:35:37 expanding transform with ExpansionRequest: components:{environments:{key:"go" value:{}}} transform:{unique_name:"External" spec:{urn:"beam:transform:org.apache.beam:schemaio_bigquery_read:v1" payload:"\nX\n\x0e\n\x08location\x1a\x02\x10\x07\n\x0c\n\x06config\x1a\x02\x10\t\n\x12\n\ndataSchema\x1a\x04\x08\x01\x10\t\x12$3fc24beb-ef0b-4fd0-b491-64340b33ca1e\x12k\x03\x01\x04\x00f\x04\x01\rbSELECT * FROM `apache-beam-testing.beam_bigquery_io_test_temp.go_bqio_it_temp_1660164801805738010`"} environment_id:"go"} namespace:"VkgVLcucsq"
14:35:37 expansion failed
14:35:37 caused by:
14:35:37 org.apache.beam.sdk.io.gcp.bigquery.BigQuerySchemaRetrievalException: Exception while trying to retrieve schema of query
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySourceDef.getBeamSchema(BigQueryQuerySourceDef.java:179)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead.expand(BigQueryIO.java:1226)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead.expand(BigQueryIO.java:747)
14:35:37 at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548)
14:35:37 at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:482)
14:35:37 at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySchemaIOProvider$BigQuerySchemaIO$1.expand(BigQuerySchemaIOProvider.java:183)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySchemaIOProvider$BigQuerySchemaIO$1.expand(BigQuerySchemaIOProvider.java:165)
14:35:37 at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548)
14:35:37 at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:499)
14:35:37 at org.apache.beam.sdk.expansion.service.ExpansionService$TransformProvider.apply(ExpansionService.java:396)
14:35:37 at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:516)
14:35:37 at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:603)
14:35:37 at org.apache.beam.model.expansion.v1.ExpansionServiceGrpc$MethodHandlers.invoke(ExpansionServiceGrpc.java:220)
14:35:37 at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
14:35:37 at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:354)
14:35:37 at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:866)
14:35:37 at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
14:35:37 at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
14:35:37 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
14:35:37 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
14:35:37 at java.lang.Thread.run(Thread.java:750)
14:35:37 Caused by: java.lang.NullPointerException: Required parameter projectId must be specified.
14:35:37 at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:907)
14:35:37 at com.google.api.client.util.Preconditions.checkNotNull(Preconditions.java:138)
14:35:37 at com.google.api.services.bigquery.Bigquery$Jobs$Insert.<init>(Bigquery.java:1737)
14:35:37 at com.google.api.services.bigquery.Bigquery$Jobs.insert(Bigquery.java:1687)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.dryRunQuery(BigQueryServicesImpl.java:473)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryQueryHelper.dryRunQueryIfNeeded(BigQueryQueryHelper.java:73)
14:35:37 at org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySourceDef.getBeamSchema(BigQueryQuerySourceDef.java:168)
14:35:37 ... 21 more
14:35:37
14:35:37
14:35:37 goroutine 646 [running]:
14:35:37 testing.tRunner.func1.2({0xeaf720, 0xc000716e40})
14:35:37 /home/jenkins/sdk/go1.18.1/src/testing/testing.go:1389 +0x24e
14:35:37 testing.tRunner.func1()
14:35:37 /home/jenkins/sdk/go1.18.1/src/testing/testing.go:1392 +0x39f
14:35:37 panic({0xeaf720, 0xc000716e40})
14:35:37 /home/jenkins/sdk/go1.18.1/src/runtime/panic.go:838 +0x207
14:35:37 github.com/apache/beam/sdks/v2/go/pkg/beam.CrossLanguage({0xc0004801a0?, 0xc0002e20a0?}, {0x104e6c0, 0x38}, {0xc00020a820?, 0xc0000c7d30?, 0x1?}, {0xc000040b80, 0xf}, 0x0, ...)
14:35:37 /home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_GoUsingJava_Dataflow_PR/src/sdks/go/pkg/beam/xlang.go:162 +0x136
14:35:37 github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/bigqueryio.Read({0xc000480120?, 0xc0002e20a0?}, {0x11bd640, 0xea8660}, {0xc0000c7e98, 0x2, 0xc000010458?})
14:35:37 /home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_GoUsingJava_Dataflow_PR/src/sdks/go/pkg/beam/io/xlang/bigqueryio/bigquery.go:158 +0x3b7
14:35:37 github.com/apache/beam/sdks/v2/go/test/integration/io/xlang/bigquery.ReadFromQueryPipeline({0xc000040b80, 0xf}, {0xc000118c60, 0x52}, {0xe007c0, 0xc000cc4540})
14:35:37 /home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_GoUsingJava_Dataflow_PR/src/sdks/go/test/integration/io/xlang/bigquery/bigquery_test.go:191 +0x2d4
14:35:37 github.com/apache/beam/sdks/v2/go/test/integration/io/xlang/bigquery.TestBigQueryIO_BasicWriteQueryRead(0xc0008b21a0)
14:35:37 /home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_GoUsingJava_Dataflow_PR/src/sdks/go/test/integration/io/xlang/bigquery/bigquery_test.go:246 +0x225
14:35:37 testing.tRunner(0xc0008b21a0, 0x107cfa0)
14:35:37 /home/jenkins/sdk/go1.18.1/src/testing/testing.go:1439 +0x102
14:35:37 created by testing.(*T).Run
14:35:37 /home/jenkins/sdk/go1.18.1/src/testing/testing.go:1486 +0x35f
14:35:37 FAIL github.com/apache/beam/sdks/v2/go/test/integration/io/xlang/bigquery 979.320s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1154750845
This seems like the best approach would be to have this extra pipeline option only read while using the GCP expansion service, but I'm not sure if there's a straightforward way to alter the behavior only when using that expansion service
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] closed pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
URL: https://github.com/apache/beam/pull/21852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1456070473
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1154750068
R: @chamikaramj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1155955275
Looks like the test is still failing, which means my local environment isn't replicating the bug. I'll look into this tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1155881796
I've changed it so that now GcpExpansionService is a derived class of the original ExpansionService, and only that class receives the project pipeline option. So only the GCP expansion service jar has grown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a diff in pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on code in PR #21852:
URL: https://github.com/apache/beam/pull/21852#discussion_r897252212
##########
sdks/java/expansion-service/build.gradle:
##########
@@ -40,6 +40,7 @@ dependencies {
implementation project(path: ":sdks:java:core", configuration: "shadow")
implementation project(path: ":runners:core-construction-java")
implementation project(path: ":runners:java-fn-execution")
+ implementation project(path: ":sdks:java:extensions:google-cloud-platform-core")
Review Comment:
Also it could create conflicts for other expansion service jars. Can we use GCP expansion service instead ?
https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/expansion-service
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1155882058
Run XVR_GoUsingJava_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1155885312
Run XVR_GoUsingJava_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a diff in pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on code in PR #21852:
URL: https://github.com/apache/beam/pull/21852#discussion_r897252403
##########
sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java:
##########
@@ -570,6 +571,8 @@ private Map<String, TransformProvider> loadRegisteredTransforms() {
protected Pipeline createPipeline() {
// TODO: [BEAM-12599]: implement proper validation
PipelineOptions effectiveOpts = PipelineOptionsFactory.create();
+ GcpOptions gcpOptions = effectiveOpts.as(GcpOptions.class);
Review Comment:
This change looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1163596926
Run XVR_GoUsingJava_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1468047917
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1211231296
Run XVR_GoUsingJava_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1324403109
@lostluck should someone take this over from @youngoli ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1155882825
Run XVR_GoUsingJava_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a diff in pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on code in PR #21852:
URL: https://github.com/apache/beam/pull/21852#discussion_r897249934
##########
sdks/java/expansion-service/build.gradle:
##########
@@ -40,6 +40,7 @@ dependencies {
implementation project(path: ":sdks:java:core", configuration: "shadow")
implementation project(path: ":runners:core-construction-java")
implementation project(path: ":runners:java-fn-execution")
+ implementation project(path: ":sdks:java:extensions:google-cloud-platform-core")
Review Comment:
I would like to avoid pulling in additional dependencies to expansion service since it will boat the jar. Why was this needed ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1163584214
Run XVR_GoUsingJava_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on a diff in pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on code in PR #21852:
URL: https://github.com/apache/beam/pull/21852#discussion_r897289343
##########
sdks/java/expansion-service/build.gradle:
##########
@@ -40,6 +40,7 @@ dependencies {
implementation project(path: ":sdks:java:core", configuration: "shadow")
implementation project(path: ":runners:core-construction-java")
implementation project(path: ":runners:java-fn-execution")
+ implementation project(path: ":sdks:java:extensions:google-cloud-platform-core")
Review Comment:
The class `GcpOptions` is in that module. If this is unnecessary please let me know, but I thought adding this as a dependency was necessary to import GcpOptions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on a diff in pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on code in PR #21852:
URL: https://github.com/apache/beam/pull/21852#discussion_r897448684
##########
sdks/java/expansion-service/build.gradle:
##########
@@ -40,6 +40,7 @@ dependencies {
implementation project(path: ":sdks:java:core", configuration: "shadow")
implementation project(path: ":runners:core-construction-java")
implementation project(path: ":runners:java-fn-execution")
+ implementation project(path: ":sdks:java:extensions:google-cloud-platform-core")
Review Comment:
Done, refactored so that only the GCP expansion service receives this pipeline option, and only that jar has increased in size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a diff in pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on code in PR #21852:
URL: https://github.com/apache/beam/pull/21852#discussion_r897311595
##########
sdks/java/expansion-service/build.gradle:
##########
@@ -40,6 +40,7 @@ dependencies {
implementation project(path: ":sdks:java:core", configuration: "shadow")
implementation project(path: ":runners:core-construction-java")
implementation project(path: ":runners:java-fn-execution")
+ implementation project(path: ":sdks:java:extensions:google-cloud-platform-core")
Review Comment:
Is see, in that case think we would need more refactoring to do this without unnecessarily bloating all expansion service jars.
For example, we could consider introducing a method to "ExpansionService" called "configureExpansionServiceProperties" (a no-op by default) and implement that in a "ExpansionService" sub-class available in "google-cloud-platform/expansion-service/"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #21852: [#21761] Add GCP Project as pipeline opt read by expansion service
Posted by GitBox <gi...@apache.org>.
youngoli commented on PR #21852:
URL: https://github.com/apache/beam/pull/21852#issuecomment-1155882409
Run Go PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org