You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Vinodraj Konda (Jira)" <ji...@apache.org> on 2022/01/20 16:57:00 UTC
[jira] [Created] (BEAM-13706) Beam IO WriteToText - FileNotFoundError
Vinodraj Konda created BEAM-13706:
-------------------------------------
Summary: Beam IO WriteToText - FileNotFoundError
Key: BEAM-13706
URL: https://issues.apache.org/jira/browse/BEAM-13706
Project: Beam
Issue Type: Bug
Components: io-py-files
Affects Versions: 2.26.0
Environment: RHEL 7 & python 3.6
Reporter: Vinodraj Konda
Hi Team,
We are running a beam pipeline & writing the final output to a file as below. This runs fine most times but fails sometimes with error "FileNotFoundError: [Errno 2] No such file or directory". This has been intermittent and there is no issue with the file location, other jobs using the same location run fine at the same time & this too works fine if we immediately re-try. Please check and advise, I couldn't find any solution/details for this online. Thanks in advance.
| 'write to file' >> beam.io.WriteToText(self.output_file,
header=','.join(output_cols_header),
shard_name_template=''
)
+*ERROR:*+
[2022-01-17 15:20:43,512] \{bash_operator.py:126} INFO - Traceback (most recent call last):
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "apache_beam/runners/common.py", line 742, in apache_beam.runners.common.PerWindowInvoker.invoke_process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "apache_beam/runners/common.py", line 867, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/iobase.py", line 1148, in process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - writer = self.sink.open_writer(init_result, str(uuid.uuid4()))
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py", line 135, in _f
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return fnc(self, *args, **kwargs)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 196, in open_writer
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return FileBasedSinkWriter(self, writer_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 417, in __init__
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - self.temp_handle = self.sink.open(temp_shard_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/textio.py", line 405, in open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - file_handle = super(_TextSink, self).open(temp_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py", line 135, in _f
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return fnc(self, *args, **kwargs)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 138, in open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return FileSystems.create(temp_path, self.mime_type, self.compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filesystems.py", line 229, in create
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return filesystem.create(path, mime_type, compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py", line 168, in create
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return self._path_open(path, 'wb', mime_type, compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py", line 143, in _path_open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - raw_file = io.open(path, mode)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - FileNotFoundError: [Errno 2] No such file or directory: '/dropbox/weather/prd/kumulus/data/earthsat/daily/2022/01/v2/beam-temp-earthsat_daily_4cast_20220117.csv-0163f65477a911ecb8d80aec9c027dc6/da21a024-9253-4500-9d10-576bdb6a4711.earthsat_daily_4cast_20220117.csv'
--
This message was sent by Atlassian Jira
(v8.20.1#820001)