You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Beam JIRA Bot (Jira)" <ji...@apache.org> on 2022/03/28 17:26:00 UTC
[jira] [Updated] (BEAM-13706) Beam IO WriteToText - FileNotFoundError
[ https://issues.apache.org/jira/browse/BEAM-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Beam JIRA Bot updated BEAM-13706:
---------------------------------
Labels: stale-P2 (was: )
> Beam IO WriteToText - FileNotFoundError
> ---------------------------------------
>
> Key: BEAM-13706
> URL: https://issues.apache.org/jira/browse/BEAM-13706
> Project: Beam
> Issue Type: Bug
> Components: io-py-files
> Affects Versions: 2.26.0
> Environment: RHEL 7 & python 3.6
> Reporter: Vinodraj Konda
> Priority: P2
> Labels: stale-P2
>
> Hi Team,
> We are running a beam pipeline & writing the final output to a file as below. This runs fine most times but fails sometimes with error "FileNotFoundError: [Errno 2] No such file or directory". This has been intermittent and there is no issue with the file location, other jobs using the same location run fine at the same time & this too works fine if we immediately re-try. Please check and advise, I couldn't find any solution/details for this online. Thanks in advance.
>
> | 'write to file' >> beam.io.WriteToText(self.output_file,
> header=','.join(output_cols_header),
> shard_name_template=''
> )
>
> +*ERROR:*+
> [2022-01-17 15:20:43,512] \{bash_operator.py:126} INFO - Traceback (most recent call last):
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "apache_beam/runners/common.py", line 742, in apache_beam.runners.common.PerWindowInvoker.invoke_process
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "apache_beam/runners/common.py", line 867, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/iobase.py", line 1148, in process
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - writer = self.sink.open_writer(init_result, str(uuid.uuid4()))
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py", line 135, in _f
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return fnc(self, *args, **kwargs)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 196, in open_writer
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return FileBasedSinkWriter(self, writer_path)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 417, in __init__
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - self.temp_handle = self.sink.open(temp_shard_path)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/textio.py", line 405, in open
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - file_handle = super(_TextSink, self).open(temp_path)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py", line 135, in _f
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return fnc(self, *args, **kwargs)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 138, in open
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return FileSystems.create(temp_path, self.mime_type, self.compression_type)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filesystems.py", line 229, in create
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return filesystem.create(path, mime_type, compression_type)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py", line 168, in create
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return self._path_open(path, 'wb', mime_type, compression_type)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py", line 143, in _path_open
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - raw_file = io.open(path, mode)
> [2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - FileNotFoundError: [Errno 2] No such file or directory: '/dropbox/weather/prd/kumulus/data/earthsat/daily/2022/01/v2/beam-temp-earthsat_daily_4cast_20220117.csv-0163f65477a911ecb8d80aec9c027dc6/da21a024-9253-4500-9d10-576bdb6a4711.earthsat_daily_4cast_20220117.csv'
--
This message was sent by Atlassian Jira
(v8.20.1#820001)