You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "liferoad (via GitHub)" <gi...@apache.org> on 2023/09/11 19:29:51 UTC
[GitHub] [beam] liferoad opened a new issue, #28399: [Bug]: The DataflowRunner behavior is changed when removing the runner v1 code
liferoad opened a new issue, #28399:
URL: https://github.com/apache/beam/issues/28399
### What happened?
The attached notebook is broken now due to #27196.
[Dataflow_Word_Count.ipynb.zip](https://github.com/apache/beam/files/12579683/Dataflow_Word_Count.ipynb.zip)
The issue is `p = beam.Pipeline(InteractiveRunner())` is used to create the pipeline but later when switching to DataflowRunner with `DataflowRunner().run_pipeline(p, options=options)`, DataflowRunner takes the Dataflow job as the legacy runner and disables the Runner V2.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on issue #28399: [Bug]: The DataflowRunner behavior is changed when removing the runner v1 code
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #28399:
URL: https://github.com/apache/beam/issues/28399#issuecomment-1721700395
I could not figure out what cause this issue. I am wondering whether we need call `_check_and_add_missing_options` somewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn closed issue #28399: [Bug]: The DataflowRunner behavior is changed when removing the runner v1 code
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #28399: [Bug]: The DataflowRunner behavior is changed when removing the runner v1 code
URL: https://github.com/apache/beam/issues/28399
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on issue #28399: [Bug]: The DataflowRunner behavior is changed when removing the runner v1 code
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #28399:
URL: https://github.com/apache/beam/issues/28399#issuecomment-1714460113
@robertwb FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on issue #28399: [Bug]: The DataflowRunner behavior is changed when removing the runner v1 code
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on issue #28399:
URL: https://github.com/apache/beam/issues/28399#issuecomment-1721629767
DataflowRunner().run_pipeline(p, options=options) should work just as well as before. Could you clarify exactly what is going wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on issue #28399: [Bug]: The DataflowRunner behavior is changed when removing the runner v1 code
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on issue #28399:
URL: https://github.com/apache/beam/issues/28399#issuecomment-1721805319
This boils down to https://s.apache.org/no-beam-pipeline :).
It looks like manually setting these flags after construction may be sufficient. Right now we're hijacking the `apply()` on every PTransform to set them, which would be good to see if we can avoid. It should also be possible to fix this service-side so the client doesn't need to pass these arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #28399: [Bug]: The DataflowRunner behavior is changed when removing the runner v1 code
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #28399:
URL: https://github.com/apache/beam/issues/28399#issuecomment-1721664056
Repro:
```
import argparse
import logging
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.runners import DataflowRunner
from apache_beam.runners.interactive.interactive_runner import InteractiveRunner
def run(argv=None):
parser = argparse.ArgumentParser()
options = PipelineOptions()
p = beam.Pipeline(InteractiveRunner())
p | beam.Create([1])
beam.runners.DataflowRunner().run_pipeline(p, options=options).wait_until_finish()
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
run()
```
```
python pipeline.py --project google.com:clouddfe --temp_location=gs://clouddfe-valentyn --staging_location=gs://clouddfe-valentyn --region us-central1
```
fails with
ERROR:apache_beam.runners.dataflow.dataflow_runner:2023-09-15T18:10:32.288Z: JOB_MESSAGE_ERROR: Runnable workflow has no steps specified.
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2023-09-15_11_10_28-12637281573769828865 is in state JOB_STATE_FAILED
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org