[jira] [Created] (BEAM-4711) LocalFileSystem.delete doesn't support globbing

["Ryan Williams (JIRA)" <ji...@apache.org>]

Ryan Williams created BEAM-4711:
-----------------------------------

             Summary: LocalFileSystem.delete doesn't support globbing
                 Key: BEAM-4711
                 URL: https://issues.apache.org/jira/browse/BEAM-4711
             Project: Beam
          Issue Type: Task
          Components: sdk-py-core
    Affects Versions: 2.5.0
            Reporter: Ryan Williams
            Assignee: Ryan Williams


I attempted to run {{wordcount_it_test:WordCountIT.test_wordcount_it}} locally with {{DirectRunner}}:

{code}
python setup.py nosetests \
  --tests apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it \
  --test-pipeline-options="--output=foo"
{code}

It failed in [the {{delete_files}} cleanup command|https://github.com/apache/beam/blob/a58f1ffaafb0e2ebcc73a1c5abfb05a15ec6a84b/sdks/python/apache_beam/examples/wordcount_it_test.py#L64]:

{code}
root: WARNING: Retry with exponential backoff: waiting for 11.1454450937 seconds before retrying delete_files because we caught exception: BeamIOError: Delete operation failed with exceptions {'foo/1530557644/results*': IOError(OSError(2, 'No such file or directory'),)}
 Traceback for above exception (most recent call last):
  File "/Users/ryan/c/beam/sdks/python/apache_beam/utils/retry.py", line 184, in wrapper
    return fun(*args, **kwargs)
  File "/Users/ryan/c/beam/sdks/python/apache_beam/testing/test_utils.py", line 136, in delete_files
    FileSystems.delete(file_paths)
  File "/Users/ryan/c/beam/sdks/python/apache_beam/io/filesystems.py", line 282, in delete
    return filesystem.delete(paths)
  File "/Users/ryan/c/beam/sdks/python/apache_beam/io/localfilesystem.py", line 304, in delete
    raise BeamIOError("Delete operation failed", exceptions)
{code}

The line:

{code}
self.addCleanup(delete_files, [output + '*'])
{code}

works as expected in GCS, and deletes a test's output-directory, but it fails in on the local-filesystem, which doesn't expand globs before attempting to delete paths.

It would be good to make these consistent, presumably by adding glob-support to {{LocalFileSystem}}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)