[jira] [Work logged] (BEAM-5417) FileSystems.match behaviour diff between GCS and local file system

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-5417?focusedWorklogId=150277&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150277 ]

ASF GitHub Bot logged work on BEAM-5417:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 02/Oct/18 01:12
            Start Date: 02/Oct/18 01:12
    Worklog Time Spent: 10m 
      Work Description: udim commented on a change in pull request #6423: [BEAM-5417] Parity between GCS and local match
URL: https://github.com/apache/beam/pull/6423#discussion_r221802288
 
 

 ##########
 File path: sdks/python/apache_beam/io/filesystem.py
 ##########
 @@ -531,24 +530,117 @@ def _list(self, dir_or_prefix):
     """
     raise NotImplementedError
 
+  @staticmethod
+  def _split_scheme(url_or_path):
+    match = re.match(r'(^[a-z]+)://(.*)', url_or_path)
+    if match is not None:
+      return match.groups()
+    return None, url_or_path
+
+  @staticmethod
+  def _combine_scheme(scheme, path):
+    if scheme is None:
+      return path
+    return '{}://{}'.format(scheme, path)
+
   def _url_dirname(self, url_or_path):
     """Like posixpath.dirname, but preserves scheme:// prefix.
 
     Args:
       url_or_path: A string in the form of scheme://some/path OR /some/path.
     """
-    match = re.match(r'([a-z]+://)(.*)', url_or_path)
-    if match is None:
-      return posixpath.dirname(url_or_path)
-    url_prefix, path = match.groups()
-    return url_prefix + posixpath.dirname(path)
+    scheme, path = self._split_scheme(url_or_path)
+    return self._combine_scheme(scheme, posixpath.dirname(path))
+
+  def match_files(self, file_metas, pattern):
+    """Filter :class:`FileMetadata` objects by :data:`pattern`
+
+    Args:
+      file_metas (:obj:`list` of :class:`FileMetadata`):
+        Files to consider when matching
+      pattern (str): File pattern
+
+    See Also:
+      :meth:`translate_pattern`
+
+    Returns:
+      Generator of matching :class:`FileMetadata`
+    """
+    re_pattern = re.compile(self.translate_pattern(pattern))
+    match = re_pattern.match
+    for file_metadata in file_metas:
+      is_match = match(file_metadata.path)
+      logger.debug('%r %r', is_match, file_metadata)
+      if is_match:
+        yield file_metadata
+
+  @staticmethod
+  def translate_pattern(pattern):
+    """
+    Translate a :data:`pattern` to a regular expression.
+    There is no way to quote meta-characters.
+
+    Pattern syntax:
+      The pattern syntax is based on the fnmatch_ syntax, with the following
+      differences:
+
+      -   ``*`` Is equivalent to ``[^/\\]*`` rather than ``.*``.
+      -   ``**`` Is equivalent to ``.*``.
+
+    See also:
+      :meth:`match` uses this method
+
+    This method is based on `Python 2.7's fnmatch.translate`_.
 
 Review comment:
   According to https://issues.apache.org/jira/browse/LEGAL-417, let's:
   - Add a note to this method saying:
   `The code in this method is licensed under PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2.`
   - Add the full text of https://github.com/python/cpython/blob/master/LICENSE to the end of https://github.com/apache/beam/blob/master/LICENSE.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 150277)
    Time Spent: 5h 10m  (was: 5h)

> FileSystems.match behaviour diff between GCS and local file system
> ------------------------------------------------------------------
>
>                 Key: BEAM-5417
>                 URL: https://issues.apache.org/jira/browse/BEAM-5417
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>    Affects Versions: 2.5.0, 2.6.0
>            Reporter: Joar Wandborg
>            Assignee: Chamikara Jayalath
>            Priority: Major
>          Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> Given the directory structure:
>  
> {noformat}
> .
> ├── filesystem-match-test
> │   ├── a
> │   │   └── file.txt
> │   └── b
> │       └── file.txt
> └── filesystem-match-test.py
> {noformat}
>  
> Where {{filesystem-match-test.py}} contains:
> {code:python}
> from __future__ import print_function
> import os
> import posixpath
> from apache_beam.io.filesystem import MatchResult
> from apache_beam.io.filesystems import FileSystems
> BASES = [
>     os.path.join(os.path.dirname(__file__), "./"),
>     "gs://my-bucket/test/",
> ]
> pattern = "filesystem-match-test/*/file.txt"
> for base_path in BASES:
>     full_pattern = posixpath.join(base_path, pattern)
>     print("full_pattern: {}".format(full_pattern))
>     match_result = FileSystems.match([full_pattern])[0]  # type: MatchResult
>     print("metadata list: {}".format(match_result.metadata_list))
> {code}
> Running {{python filesystem-match-test.py}} does not match any files locally, but does match files on GCS:
> {noformat}
> full_pattern: ./filesystem-match-test/*/file.txt
> metadata list: []
> full_pattern: gs://my-bucket/test/filesystem-match-test/*/file.txt
> metadata list: [FileMetadata(gs://my-bucket/test/filesystem-match-test/a/file.txt, 6), FileMetadata(gs://my-bucket/test/filesystem-match-test/b/file.txt, 6)]
> {noformat}
> The expected result is that a/file.txt and b/file.txt should be matched for both patterns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)