You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2025/10/22 14:22:46 UTC
[I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
ahmedabu98 opened a new issue, #33074:
URL: https://github.com/apache/beam/issues/33074
### What happened?
I noticed this when testing with Managed BigQueryIO in Python SDK:
```python
options = PipelineOptions([
"--project=google.com:clouddfe",
"--region=us-central1",
"--temp_location=gs://ahmedabualsaud-wordcount/tmp",
])
with beam.Pipeline(options=options) as p:
(p
| beam.Create([beam.Row(int=999)])
| beam.managed.Write("bigquery",
config={"table": "google.com:clouddfe.ahmedabualsaud.test_table"},
expansion_service="localhost:12345"
))
```
I would expect these pipeline options to be picked up on Java's side. With some logging, I see the Java SDK recognizing the options at expand time in PTransforms (i.e. expand()), but not at runtime in DoFns (i.e. @ProcessElement).
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3549544866
My understanding is that the same set of pipeline options are available to all SDKs via UW but specific options might be not available due to the differences in naming conventions. I haven't tried accessing options via DoFn's process context though and it's possible that some options get dropped when accessing them via that path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3433328818
I can only reproduce this issue on Dataflow when i enable managed Transforms with --experiments=enable_managed_transforms, which forces the ManagedTransforms upgrade. when you run this on local or with out managedTransforms upgrade step. This issue is not reproduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3549102081
I think the original issue here is not solved right? A Java DoFn's process context does not include pipeline options passed thru the Python SDK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3549536013
> A Java DoFn's process context does not include pipeline options passed thru the Python SDK
I remember I checked the pipeline option is honored. The NPE in the example in description was due to other issue (https://github.com/apache/beam/issues/33074#issuecomment-3304740621)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3549541072
Yes the main issue is still not fixed. As i mentioned, In this code we pass pipeline options through PCollection Input(which i think is a rare code pattern) and Runner Context. The first path is not working, we fall back to the second path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3432627108
This is still relevant and failing Beam 2.69.0rc3: https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/2907#issuecomment-3432610269
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #33074: [Bug]: Cross-language pipeline options are not picked up in Java DoFns
URL: https://github.com/apache/beam/issues/33074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3433584265
> Also, is it expected for Beam YAML that not providing enable_managed_transforms experiments, managed transforms are still being processed?
This experiment does not have to be specified by end users and is on by default unless you run using a dev/SNAPSHOT Beam version.
Also, the underlying issue here is managed switching YAML to use BQ batch loads for batch pipelines (instead of storage write API) which requires Tarun's fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "Suvrat1629 (via GitHub)" <gi...@apache.org>.
Suvrat1629 commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-2706491261
@ahmedabu98 I am trying to understand how this would work. So do i tweak the `expand()` method and the "@ProcessElement" to pass it to the DoFns or is there some other way to do this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 closed issue #33074: [Bug]: Cross-language pipeline options are not picked up in Java DoFns
URL: https://github.com/apache/beam/issues/33074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3573536407
@tarun-google @chamikaramj, is this a release blocker or could we move this to the next release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3304504795
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #33074: [Bug]: Cross-language pipeline options are not picked up in Java DoFns
URL: https://github.com/apache/beam/issues/33074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3434258509
> I see. Can we make sure that we release managed config before switching templates to the new version ? Also, as a workaround you can provide the pipeline option `--updateCompatibilityVersion=2.68.0` which will prevent YAML from switching to managed.
Confirmed that adding `--update_compatibility_version=2.68.0` pipeline option worked for 2.69.0rc3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3433335330
cc @chamikaramj are pipeline options not included in the graph that were sending to the DF service? or is the service somehow dropping the options?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3262085490
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3433375913
> I can only reproduce this issue on Dataflow when i enable managed Transforms with --experiments=enable_managed_transforms, which forces the ManagedTransforms upgrade. when you run this on local or with out managedTransforms upgrade step. This issue is not reproduced.
It could be a related issue, but the jobs in https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/2907 doesn't have managed transform upgrade enabled (no ` --experiments=enable_managed_transforms`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3433518455
I see. Can we make sure that we get the managed fix in before switching templates to the new version ? Also, as a workaround you can provide the pipeline option `--updateCompatibilityVersion=2.68.0` which will prevent YAML from switching to managed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3160028853
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3576934049
I don't think this is a release blocker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3433315335
> Firstly, the issue is only in the ManagedTransforms path. PipelineOptions are not passed to [PCollection input](https://github.com/apache/beam/blob/d6422d1277ad063a5991d78c25fa8fe209ad2dd4/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryFileLoadsSchemaTransformProvider.java#L93) in the managedTransforms path.
@tarun-google can you elaborate? I just ran a pipeline and printed out the options at that particular line, and I see all the options as expected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3433504290
I may understand what happened now. Managed transform upgrade overrides the BigQueryIO from 2.69.0rc3->2.68, which did not have the fix #36209 and thus caused the issue.
@chamikaramj Although this is WAI, the implication is after Beam 2.69.0 released and before managed transform configuration changed to use Beam 2.69.0, managed BQIO will break for YAML Templates
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3304740621
Cross language pipeline options are not picked up inside DoFns as expansion service do not exist at runtime. When users provide temp_location, during the pipeline construction expansion service creates PipelineOptions with(tempLocation). But when we directly access at runtime, it exists as temp_Location, as expansion service is not doing cross language re-naming.
The usage of pipeline options inside DoFn is not considered as legal pattern. So, I will be making a fix localized to BigQueryIO to access temp_Location. This will allow ManagedIO upgrade not to break existing BQ python/yaml pipelines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3577657853
Possibly relevant - https://github.com/apache/beam/pull/36443 fixed passing pipeline options from Python to Java through external transforms, which may have been the initial issue (sounds like we have more to do here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3577414950
Sounds good. Will move it to the next release then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cross-language pipeline options are not picked up in Java DoFns [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on issue #33074:
URL: https://github.com/apache/beam/issues/33074#issuecomment-3433473694
But still given the fix we made to allow using temp_location as backup on run-time. This should not NPE create issue in Templates as long as temp_location is passed properly. At least integration tests we fixed in Beam shows that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org