You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "chamikaramj (via GitHub)" <gi...@apache.org> on 2024/11/05 22:34:03 UTC
[PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
chamikaramj opened a new pull request, #33026:
URL: https://github.com/apache/beam/pull/33026
This fixes https://github.com/apache/beam/issues/33022.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #33026:
URL: https://github.com/apache/beam/pull/33026#issuecomment-2466533417
Java PipelineOptions deserialization expects the encoded options to have a specific key.
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/PipelineOptionsTranslation.java#L99
```
"beam:option:" + snake_case_options_key + ":v1"
```
May be options forwarded from Python are not in this format ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #33026:
URL: https://github.com/apache/beam/pull/33026#discussion_r1831444171
##########
sdks/java/expansion-service/src/test/java/org/apache/beam/sdk/expansion/service/ExpansionServiceTest.java:
##########
@@ -146,6 +206,58 @@ public void testConstruct() {
}
}
+ @Test
+ public void testConstructWithPipelineOptions() {
+ PipelineOptionsFactory.register(TestOptions.class);
+ Pipeline p = Pipeline.create();
+ p.apply(Impulse.create());
+ RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(p);
+ String inputPcollId =
+ Iterables.getOnlyElement(
+ Iterables.getOnlyElement(pipelineProto.getComponents().getTransformsMap().values())
+ .getOutputsMap()
+ .values());
+
+ Struct optionsStruct =
+ Struct.newBuilder()
+ .putFields(
+ PIPELINE_OPTIONS_URN_PREFIX + "string_option" + PIPELINE_OPTIONS_URN_SUFFIX,
+ Value.newBuilder()
+ .setStringValue(TestTransformRegistrar.EXPECTED_STRING_VALUE)
+ .build())
+ .putFields(
+ PIPELINE_OPTIONS_URN_PREFIX + "boolean_option" + PIPELINE_OPTIONS_URN_SUFFIX,
+ Value.newBuilder()
+ .setBoolValue(TestTransformRegistrar.EXPECTED_BOOLEAN_VALUE)
+ .build())
+ .putFields(
+ PIPELINE_OPTIONS_URN_PREFIX + "integer_option" + PIPELINE_OPTIONS_URN_SUFFIX,
+ Value.newBuilder()
+ .setNumberValue(TestTransformRegistrar.EXPECTED_INTEGER_VALUE)
+ .build())
+ .build();
+ ExpansionApi.ExpansionRequest request =
+ ExpansionApi.ExpansionRequest.newBuilder()
+ .setComponents(pipelineProto.getComponents())
+ .setPipelineOptions(optionsStruct)
+ .setTransform(
+ RunnerApi.PTransform.newBuilder()
+ .setUniqueName(TEST_NAME)
+ .setSpec(RunnerApi.FunctionSpec.newBuilder().setUrn(TEST_OPTIONS_URN))
+ .putInputs("input", inputPcollId))
+ .setNamespace(TEST_NAMESPACE)
+ .build();
+ ExpansionApi.ExpansionResponse response = expansionService.expand(request);
Review Comment:
Yeah, the `expansionService.expand` call results in the transform provided in the `ExpansionRequest` being expanded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #33026:
URL: https://github.com/apache/beam/pull/33026#issuecomment-2466002493
This is supposed to work for cross-language too right? I tried the following pipeline but the underlying transform doesn't recognize the pipeline options passed from the Python SDK:
```python
options = PipelineOptions([
"--project=google.com:clouddfe",
"--region=us-central1",
"--temp_location=gs://ahmedabualsaud-wordcount/tmp",
])
with beam.Pipeline(options=options) as p:
(p
| beam.Create([beam.Row(int=999)])
| beam.managed.Write("bigquery",
config={"table": "google.com:clouddfe.ahmedabualsaud.test_table"}))
```
File Loads is meant to fetch the GCS location from pipeline options `temp_location`, but does not find it. I added some logging and can see that pipeline options are not getting picked up by the expanded xlang transform
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj merged PR #33026:
URL: https://github.com/apache/beam/pull/33026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #33026:
URL: https://github.com/apache/beam/pull/33026#discussion_r1831301201
##########
sdks/java/expansion-service/src/test/java/org/apache/beam/sdk/expansion/service/ExpansionServiceTest.java:
##########
@@ -146,6 +206,58 @@ public void testConstruct() {
}
}
+ @Test
+ public void testConstructWithPipelineOptions() {
+ PipelineOptionsFactory.register(TestOptions.class);
+ Pipeline p = Pipeline.create();
+ p.apply(Impulse.create());
+ RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(p);
+ String inputPcollId =
+ Iterables.getOnlyElement(
+ Iterables.getOnlyElement(pipelineProto.getComponents().getTransformsMap().values())
+ .getOutputsMap()
+ .values());
+
+ Struct optionsStruct =
+ Struct.newBuilder()
+ .putFields(
+ PIPELINE_OPTIONS_URN_PREFIX + "string_option" + PIPELINE_OPTIONS_URN_SUFFIX,
+ Value.newBuilder()
+ .setStringValue(TestTransformRegistrar.EXPECTED_STRING_VALUE)
+ .build())
+ .putFields(
+ PIPELINE_OPTIONS_URN_PREFIX + "boolean_option" + PIPELINE_OPTIONS_URN_SUFFIX,
+ Value.newBuilder()
+ .setBoolValue(TestTransformRegistrar.EXPECTED_BOOLEAN_VALUE)
+ .build())
+ .putFields(
+ PIPELINE_OPTIONS_URN_PREFIX + "integer_option" + PIPELINE_OPTIONS_URN_SUFFIX,
+ Value.newBuilder()
+ .setNumberValue(TestTransformRegistrar.EXPECTED_INTEGER_VALUE)
+ .build())
+ .build();
+ ExpansionApi.ExpansionRequest request =
+ ExpansionApi.ExpansionRequest.newBuilder()
+ .setComponents(pipelineProto.getComponents())
+ .setPipelineOptions(optionsStruct)
+ .setTransform(
+ RunnerApi.PTransform.newBuilder()
+ .setUniqueName(TEST_NAME)
+ .setSpec(RunnerApi.FunctionSpec.newBuilder().setUrn(TEST_OPTIONS_URN))
+ .putInputs("input", inputPcollId))
+ .setNamespace(TEST_NAMESPACE)
+ .build();
+ ExpansionApi.ExpansionResponse response = expansionService.expand(request);
Review Comment:
Does the expansion service expand the requested transform? Trying to follow how the assertions in `TestOptionsTransform`'s expand are reached.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #33026:
URL: https://github.com/apache/beam/pull/33026#issuecomment-2468070983
Nevermind ignore my comments. The options are getting through to Java, but are somehow not getting picked up in step in the IO. I think the changes in this PR are working though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #33026:
URL: https://github.com/apache/beam/pull/33026#issuecomment-2468011161
Python translation does put them in this format: https://github.com/apache/beam/blob/2488ca131bed3f28c92fa7b38b7506a461818a3a/sdks/python/apache_beam/options/pipeline_options.py#L427-L433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #33026:
URL: https://github.com/apache/beam/pull/33026#issuecomment-2468166146
Looks like it's a general Java issue. Opened #33074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #33026:
URL: https://github.com/apache/beam/pull/33026#issuecomment-2458274589
R: @ahmedabu98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33026:
URL: https://github.com/apache/beam/pull/33026#issuecomment-2458278143
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #33026:
URL: https://github.com/apache/beam/pull/33026#issuecomment-2462966148
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #33026:
URL: https://github.com/apache/beam/pull/33026#issuecomment-2463128171
Cdap failure in "Java IOs Direct" is unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Java ExpansionService to use arbitrary PipelineOptions set through an ExpansionRequest [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #33026:
URL: https://github.com/apache/beam/pull/33026#issuecomment-2468043947
The issue might be that pipeline options are ignored when building the Java SchemaTransform: https://github.com/apache/beam/blob/2488ca131bed3f28c92fa7b38b7506a461818a3a/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionServiceSchemaTransformProvider.java#L94C69-L94C76
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org