You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 03:55:39 UTC
[GitHub] [beam] kennknowles opened a new issue, #19537: FileBasedSink is not creating file paths according to target filesystem
kennknowles opened a new issue, #19537:
URL: https://github.com/apache/beam/issues/19537
File path generated in _open_writer_ method is not according to target filesystem, because
os.path.join is used and not FileSystems.join.
apache_beam\io\filebasedsink.py extract:
```
def _create_temp_dir(self, file_path_prefix):
base_path, last_component = FileSystems.split(file_path_prefix)
if not last_component:
# Trying to re-split the base_path to check if it's a root.
new_base_path,
_ = FileSystems.split(base_path)
if base_path == new_base_path:
raise ValueError('Cannot create
a temporary directory for root path '
'prefix %s. Please specify a file path
prefix with '
'at least two components.' % file_path_prefix)
path_components
= [base_path,
'beam-temp-' + last_component + '-' + uuid.uuid1().hex]
return
FileSystems.join(*path_components)
@check_accessible(['file_path_prefix', 'file_name_suffix'])
def
open_writer(self, init_result, uid):
# A proper suffix is needed for AUTO compression detection.
# We also ensure there will be no collisions with uid and a
# (possibly unsharded) file_path_prefix
and a (possibly empty)
# file_name_suffix.
file_path_prefix = self.file_path_prefix.get()
file_name_suffix
= self.file_name_suffix.get()
suffix = (
'.' + os.path.basename(file_path_prefix) + file_name_suffix)
return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
```
This created incompatibilities between, for example, Windows and GCS.
Expected: gs://bucket/beam-temp-result-uuid\\uid.result
Actual: gs://bucket/beam-temp-result-uuid/uid.result
Replacing os.path.join with FileSystems.join fixes the issue
Imported from Jira [BEAM-6821](https://issues.apache.org/jira/browse/BEAM-6821). Original Jira may contain additional context.
Reported by: gkovelman.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] FileBasedSink is not creating file paths according to target filesystem [beam]
Posted by "damccorm (via GitHub) via github" <gi...@beam.apache.org>.
damccorm commented on issue #19537:
URL: https://github.com/apache/beam/issues/19537#issuecomment-6749105897
Resolved by #8054 (commit 13fcf7851a), which changed `FileBasedSink.open_writer` to use `FileSystems.join` instead of `os.path.join`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] FileBasedSink is not creating file paths according to target filesystem [beam]
Posted by "damccorm (via GitHub) via github" <gi...@beam.apache.org>.
damccorm closed issue #19537: FileBasedSink is not creating file paths according to target filesystem
URL: https://github.com/apache/beam/issues/19537
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] FileBasedSink is not creating file paths according to target filesystem [beam]
Posted by "physicalist (via GitHub)" <gi...@apache.org>.
physicalist commented on issue #19537:
URL: https://github.com/apache/beam/issues/19537#issuecomment-3908228342
Seems to already have been fixed in 2019 in https://github.com/apache/beam/commit/13fcf7851a1e68e3078a924df3e845ecb42bdf3c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org