You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 22:39:48 UTC
[GitHub] [beam] damccorm opened a new issue, #21319: Beam IO WriteToText - FileNotFoundError
damccorm opened a new issue, #21319:
URL: https://github.com/apache/beam/issues/21319
Hi Team,
We are running a beam pipeline & writing the final output to a file as below. This runs fine most times but fails sometimes with error "FileNotFoundError: [Errno 2] No such file or directory". This has been intermittent and there is no issue with the file location, other jobs using the same location run fine at the same time & this too works fine if we immediately re-try. Please check and advise, I couldn't find any solution/details for this online. Thanks in advance.
| 'write to file' \>\> beam.io.WriteToText(self.output_file,
header=','.join(output_cols_header),
shard_name_template=''
)
*****ERROR:*****
[2022-01-17 15:20:43,512] \{bash_operator.py:126} INFO - Traceback (most recent call last):
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "apache_beam/runners/common.py", line 742, in apache_beam.runners.common.PerWindowInvoker.invoke_process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "apache_beam/runners/common.py", line 867, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/iobase.py", line 1148, in process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - writer = self.sink.open_writer(init_result, str(uuid.uuid4()))
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py", line 135, in _f
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return fnc(self, *args, **kwargs)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 196, in open_writer
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return FileBasedSinkWriter(self, writer_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 417, in __init__
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - self.temp_handle = self.sink.open(temp_shard_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/textio.py", line 405, in open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - file_handle = super(_TextSink, self).open(temp_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py", line 135, in _f
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return fnc(self, *args, **kwargs)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 138, in open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return FileSystems.create(temp_path, self.mime_type, self.compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filesystems.py", line 229, in create
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return filesystem.create(path, mime_type, compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py", line 168, in create
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return self._path_open(path, 'wb', mime_type, compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py", line 143, in _path_open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - raw_file = io.open(path, mode)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - FileNotFoundError: [Errno 2] No such file or directory: '/dropbox/weather/prd/kumulus/data/earthsat/daily/2022/01/v2/beam-temp-earthsat_daily_4cast_20220117.csv-0163f65477a911ecb8d80aec9c027dc6/da21a024-9253-4500-9d10-576bdb6a4711.earthsat_daily_4cast_20220117.csv'
Imported from Jira [BEAM-13706](https://issues.apache.org/jira/browse/BEAM-13706). Original Jira may contain additional context.
Reported by: vinodraj.konda.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org