You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Bob Rudis (Jira)" <ji...@apache.org> on 2022/01/07 11:00:00 UTC

[jira] [Created] (ARROW-15280) Ignore "*_$folder$" files on S3 in {arrow} R package

Bob Rudis created ARROW-15280:
---------------------------------

             Summary: Ignore "*_$folder$" files on S3 in {arrow} R package
                 Key: ARROW-15280
                 URL: https://issues.apache.org/jira/browse/ARROW-15280
             Project: Apache Arrow
          Issue Type: Improvement
          Components: R
    Affects Versions: 6.0.1
            Reporter: Bob Rudis


ARROW-4406 notes that:

>Currently reading parquet files generated by Hadoop (EMR) from S3 fails with "ValueError: >Found files in an intermediate directory" because of the _$folder$ empty files.

This was fixed in the pyarrow but R still has this issue.

The R side does not seem to have something similar to:

{{  def _should_silently_exclude(self, file_name):}}
{{    return (file_name.endswith('.crc') or # Checksums}}
{{            file_name.endswith('_$folder$') or # HDFS directories in S3}}
{{            file_name.startswith('.') or # Hidden files starting with .}}
{{            file_name.startswith('_') or # Hidden files starting with _}}
{{            file_name in EXCLUDED_PARQUET_PATHS)}}





--
This message was sent by Atlassian Jira
(v8.20.1#820001)