You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/03/14 08:57:00 UTC
[jira] [Work logged] (BEAM-6821) FileBasedSink is not creating file
paths according to target filesystem
[ https://issues.apache.org/jira/browse/BEAM-6821?focusedWorklogId=213002&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-213002 ]
ASF GitHub Bot logged work on BEAM-6821:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Mar/19 08:56
Start Date: 14/Mar/19 08:56
Worklog Time Spent: 10m
Work Description: gkovelman commented on pull request #8054: [BEAM-6821] FileBasedSink improper paths
URL: https://github.com/apache/beam/pull/8054
File path generated in open_writer method is not according to target filesystem, because os.path.join is used and not FileSystems.join.
This created incompatibilities between, for example, Windows and GCS.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) <br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 213002)
Time Spent: 10m
Remaining Estimate: 0h
> FileBasedSink is not creating file paths according to target filesystem
> -----------------------------------------------------------------------
>
> Key: BEAM-6821
> URL: https://issues.apache.org/jira/browse/BEAM-6821
> Project: Beam
> Issue Type: Bug
> Components: io-java-text
> Affects Versions: 2.11.0
> Environment: Windows 10
> Reporter: Gregory Kovelman
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> File path generated in _open_writer_ method is not according to target filesystem, because
> os.path.join is used and not FileSystems.join.
> apache_beam\io\filebasedsink.py extract:
>
> {code:java}
> def _create_temp_dir(self, file_path_prefix):
> base_path, last_component = FileSystems.split(file_path_prefix)
> if not last_component:
> # Trying to re-split the base_path to check if it's a root.
> new_base_path, _ = FileSystems.split(base_path)
> if base_path == new_base_path:
> raise ValueError('Cannot create a temporary directory for root path '
> 'prefix %s. Please specify a file path prefix with '
> 'at least two components.' % file_path_prefix)
> path_components = [base_path,
> 'beam-temp-' + last_component + '-' + uuid.uuid1().hex]
> return FileSystems.join(*path_components)
> @check_accessible(['file_path_prefix', 'file_name_suffix'])
> def open_writer(self, init_result, uid):
> # A proper suffix is needed for AUTO compression detection.
> # We also ensure there will be no collisions with uid and a
> # (possibly unsharded) file_path_prefix and a (possibly empty)
> # file_name_suffix.
> file_path_prefix = self.file_path_prefix.get()
> file_name_suffix = self.file_name_suffix.get()
> suffix = (
> '.' + os.path.basename(file_path_prefix) + file_name_suffix)
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> {code}
>
>
> This created incompatibilities between, for example, Windows and GCS.
> Expected: gs://bucket/beam-temp-result-uuid\\uid.result
> Actual: gs://bucket/beam-temp-result-uuid/uid.result
> Replacing os.path.join with FileSystems.join fixes the issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)