[GitHub] [beam] damccorm opened a new issue, #21313: fileio calls mkdirs with basepath and not dirname(full_file_name)

[GitBox <gi...@apache.org>]

damccorm opened a new issue, #21313:
URL: https://github.com/apache/beam/issues/21313

   When calling `apache_beam.io.fileio.WriteToFiles` with a `file_naming` argument that adds a directory to the path, the current implementation fails to write files if a `mkdirs` or analogous call is needed in the underlying file storage.
   
   Example, 
   
   ```
   
   apache_beam.io.fileio.WriteToFiles(
       path="some/base/dir", sink=..., destination=lambda x: "events",
   
      file_naming=lambda *x: "subdir/file.txt"
   ) 
   
   ```
   
   
   the current fileio implementation will call `mkdirs` with `some/base/dir` instead of `some/base/dir/subdir`.
   
   The bug is currently at https://github.com/apache/beam/blob/67bcf1e16e3fdf68cdea7a4b42b9c003e4b8948c/sdks/python/apache_beam/io/fileio.py#L605.
   
   \==\==
   
   Personally, I would recommend changing the FileSystems interface to have `open` call `mkdirs` in storages that require root parent directory creation.
   
   Imported from Jira [BEAM-13493](https://issues.apache.org/jira/browse/BEAM-13493). Original Jira may contain additional context.
   Reported by: txomon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org