You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Ox0400 (via GitHub)" <gi...@apache.org> on 2025/05/20 11:19:30 UTC
[I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Ox0400 opened a new issue, #35013:
URL: https://github.com/apache/beam/issues/35013
### What happened?
The main points of my issue are as follows:

When submitting a DataFlowPythonOP task, the script is executed using subprocess. However, apache_beam.runners.dataflow.internal.apiclient does not adjust the default log level, and logging.info() is used to print the Job ID and URL. As a result, create_python_job fails to match the Job ID, causing the task to fail.
Although it’s possible to forcibly set the log level in the script using:
```
logging.getLogger('apache_beam.runners.dataflow.internal.apiclient').setLevel(logging.INFO)
```
this is not a very user-friendly solution.
Additional Context:
Vertex Pipeline needs to match the submitted Job URL from subprocess when running DataFlowPythonOP to call create_python_job. However, since the module does not change the default log level, the Job URL is not displayed, leading to a RuntimeError.
```
# run_pipeline.py
from google_cloud_pipeline_components.v1.dataflow import DataflowPythonJobOp
...
@dsl.pipeline(name='test dataflow')
def test_dataflow():
dataflow_task = DataflowPythonJobOp(
project=project,
location=region,
python_module_path=dataflow_clean_local_path,
requirements_file_path=requirements_file_path,
temp_location=temp_location,
args=[
"--project", project,
"--region", region,
"--temp_location", temp_location,
"--job_name", f"dataflow-clean-{time.strftime('%Y%m%d-%H%M%S', time.gmtime())}",
"--save_main_session",
"--runner", "DataflowRunner",
],
)
...
```
```
# data_clean.py
p = beam.Pipeline(options=options)
p | .....
result = p.run()
result.wait_until_finish()
return result
```
PR: https://github.com/apache/beam/pull/34952
Let me know if you'd like any refinements!
### Issue Priority
Priority: 1 (data loss / total loss of function)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [x] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [x] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2932361020
then, would fix the kubeflow code. It feels like to not the right approach to modify the logging for all users without giving them control to change default + current apiclient logs may be spammy for some users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2931477309
> have you considered the following to extract Dataflow Job ID:
>
> ```
> from apache_beam.runners.dataflow.dataflow_runner import DataflowPipelineResult
> p = beam.Pipeline(argv=pipeline_args)
> p | <...>
> result = p.run()
> if isinstance(result, DataflowPipelineResult):
> print(result.job_id())
> result.wait_until_finish()
> ```
They cannot do that since this code is completely controlled by the end users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2908069620
`data_clean.py` returns result, which should contian the job information like https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L717. Relying on the logs is not good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "Ox0400 (via GitHub)" <gi...@apache.org>.
Ox0400 commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2908695423
@liferoad Try can review this source code `dataflow_python_job_remote_runner.py`.
https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/container/v1/dataflow/dataflow_python_job_remote_runner.py
```
for line in sub_process.read_lines():
logging.info('DataflowRunner output: %s', line)
job_id, location = extract_job_id_and_location(line)
if job_id:
logging.info('Found job id %s and location %s.', job_id, location)
...
break
if not job_id:
raise RuntimeError(
'No dataflow job was found when running the python file.'
)
```
https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/container/v1/dataflow/dataflow_python_job_remote_runner.py#L33
https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/container/v1/dataflow/dataflow_python_job_remote_runner.py#L139
https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/container/v1/dataflow/dataflow_python_job_remote_runner.py#L106
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "Ox0400 (via GitHub)" <gi...@apache.org>.
Ox0400 commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-3226652859
I apologize for the delay in responding. After careful consideration, I'm not inclined to fix this bug by adding extensive new code or features. Since we haven't reached a consensus, I'll be closing this pull request along with the related issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2940861704
A compromise solution:
- Add a apache_beam.runners/dataflow/internal/dataflow_console_logs.py
- Configure the logger in that file to use INFO by default, unless overriden by the user (can compare whether _LOGGER.level == logging.UNSET)
- emit very limited, but most important logs from datflow_console_logs.py
Then, they are printed by default, if anyone doesn't like the logs, they can override the logger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2916338539
After we add `LoggingOptions`, when calling `prepare_cmd` in `google_cloud_pipeline_components`, we can pass in "--log_level=INFO" just for Dataflow jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2931417366
have you considered the following to extract Dataflow Job ID:
```
from apache_beam.runners.dataflow.dataflow_runner import DataflowPipelineResult
p = beam.Pipeline(argv=pipeline_args)
p | <...>
result = p.run()
if isinstance(result, DataflowPipelineResult):
print(result.job_id())
result.wait_until_finish()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2895170648
Can we set the log level inside DataflowPythonJobOp?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2895550883
> Can we set the log level inside DataflowPythonJobOp?
Probably not, since the user pipeline (i.e. data_clean.py) is executed through a subprocess. Setting the log level in DataflowPythonJobOp has no effect on that.
Could we just set the logging level in data_clean.py? It feels inappropriate to use warning level (as proposed in #34952) for the job id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2932358928
ack, thanks.
I think I would make this option https://github.com/apache/beam/blob/9a7914b81d2afc595c3c293ba58b1d249f9b4d7d/sdks/python/apache_beam/options/pipeline_options.py#L1333 more generic (and keep current option as alias for backwards compatibility), but also respect the content in job submission codepath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "Ox0400 (via GitHub)" <gi...@apache.org>.
Ox0400 commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2896874107
You're absolutely right—it's impossible to set the log level in DataflowPythonJobOp. The only workaround is to configure it in data_clean.py, which is what I did by adding:
logging.getLogger('apache_beam.runners.dataflow.internal.apiclient').setLevel(logging.INFO)
However, this solution is far from ideal:
Manual Overhead – Every script must explicitly set this, adding unnecessary boilerplate.
Undocumented Pitfall – There’s zero documentation warning about this requirement. Debugging this trivial issue took me days, only to discover it was caused by insufficient log levels preventing the parent process from retrieving the job ID.
Key Flaws in Current Design:
Poor Defaults – Logs critical for debugging (e.g., job IDs) should always be visible by default, not hidden behind manual configuration.
Fragile UX – Even if documented, users might overlook this step, leading to avoidable failures. Adding a "warning" for a single line of code feels like a band-aid rather than a proper fix.
Suggested Improvement:
The framework should auto-enable INFO logs for critical components (like job tracking) or fail fast with clear errors if log filtering blocks essential data. Silent failures due to log levels are a developer nightmare.
## Certainly, there are two other options:
- 1, Use print() directly.
- 2, Set the log level of the current file's `LOGGER` to `INFO`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "Ox0400 (via GitHub)" <gi...@apache.org>.
Ox0400 commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2911278512
> I see. Is this possible to always change the logging level to info like
>
> ```
> with open(python_file_path, 'r') as f:
> original_content = f.read()
>
> logging_setup_code = (
> "import logging\n"
> "logging.basicConfig(level=logging.INFO)\n\n"
> )
>
> modified_content = logging_setup_code + original_content
> # write this back to python_file_path
> ```
I understand your way, but these are two different libraries, and intrusively modifying user-submitted source files in the top library may not be a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "Ox0400 (via GitHub)" <gi...@apache.org>.
Ox0400 commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2938276871
> After we add `LoggingOptions`, when calling `prepare_cmd` in `google_cloud_pipeline_components`, we can pass in "--log_level=INFO" just for Dataflow jobs.
I see your perspective. By default, printing this log message(Job URL) when developers don't add parameters would improve the user experience, more user-friendly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2912625930
How about introducing a new logging option to control the overall logging level, e.g.,
```
from apache_beam.options.pipeline_options import PipelineOptions
class LoggingOptions(PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
parser.add_argument(
'--log_level',
default='WARNING',
help='Set the logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "Ox0400 (via GitHub)" <gi...@apache.org>.
Ox0400 commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2914725437
> How about introducing a new logging option to control the overall logging level, e.g.,
>
> ```
> from apache_beam.options.pipeline_options import PipelineOptions
>
> class LoggingOptions(PipelineOptions):
> @classmethod
> def _add_argparse_args(cls, parser):
> parser.add_argument(
> '--log_level',
> default='WARNING',
> help='Set the logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)')
> ```
This is a very good suggestion. It effectively controls the logs of all modules and provides a control interface.
However, there is an issue: when using `google_cloud_pipeline_components` to create tasks with `DataflowPythonJobOp` and run workflows, the default log level (log_level=WARNING) might not be user-friendly. It is necessary to specifically remind users to set the log level in the documentation of `google_cloud_pipeline_components`. Alternatively, the default parameters of `DataflowPythonJobOp` need to be modified.
Is it permissible to forcefully print logs such as the Job URL ? I mean that such logs can be printed under the default behavior. https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L897
```
from google_cloud_pipeline_components.v1.dataflow import DataflowPythonJobOp
...
@dsl.pipeline(name='test dataflow')
def test_dataflow():
dataflow_task = DataflowPythonJobOp(
project=project,
location=region,
python_module_path=dataflow_clean_local_path,
requirements_file_path=requirements_file_path,
temp_location=temp_location,
args=[
"--project", project,
"--region", region,
"--log_level", "INFO", # Default is WARNING, Require set INFO or DEBUG.
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #35013:
URL: https://github.com/apache/beam/issues/35013#issuecomment-2909561964
I see. Is this possible to always change the logging level to info like
```
with open(python_file_path, 'r') as f:
original_content = f.read()
logging_setup_code = (
"import logging\n"
"logging.basicConfig(level=logging.INFO)\n\n"
)
modified_content = logging_setup_code + original_content
# write this back to python_file_path
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: RuntimeError: No dataflow job was found when running the python file. [beam]
Posted by "Ox0400 (via GitHub)" <gi...@apache.org>.
Ox0400 closed issue #35013: [Bug]: RuntimeError: No dataflow job was found when running the python file.
URL: https://github.com/apache/beam/issues/35013
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org