You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Dewey Dunnington (Jira)" <ji...@apache.org> on 2022/01/31 15:00:00 UTC
[jira] [Commented] (ARROW-15280) [R] Ignore "*_$folder$" files on S3 in {arrow} R package
[ https://issues.apache.org/jira/browse/ARROW-15280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17484729#comment-17484729 ]
Dewey Dunnington commented on ARROW-15280:
------------------------------------------
Thank you for reporting this!
Where this lives in Python: https://github.com/apache/arrow/blob/ad073b7c0fec80ce88aaf1e7d6a78104711952f2/python/pyarrow/parquet.py#L1181-L1186
We don' have something similar to this approach in R, but it looks like these are exposed in the C++ API as well:
https://github.com/apache/arrow/blob/f8661e032902a963b0a6a46077d72e804d22560d/cpp/src/arrow/dataset/discovery.h#L146-L195
...which we could wire up to R here:
https://github.com/apache/arrow/blob/03219e21b42f17294fba3b3d2b22a9117fe0f080/r/src/dataset.cpp#L171-L172
and here:
https://github.com/apache/arrow/blob/03219e21b42f17294fba3b3d2b22a9117fe0f080/r/R/dataset.R#L181
> [R] Ignore "*_$folder$" files on S3 in {arrow} R package
> --------------------------------------------------------
>
> Key: ARROW-15280
> URL: https://issues.apache.org/jira/browse/ARROW-15280
> Project: Apache Arrow
> Issue Type: Improvement
> Components: R
> Affects Versions: 6.0.1
> Reporter: Bob Rudis
> Priority: Minor
>
> ARROW-4406 notes that:
> >Currently reading parquet files generated by Hadoop (EMR) from S3 fails with "ValueError: >Found files in an intermediate directory" because of the _$folder$ empty files.
> This was fixed in the pyarrow but R still has this issue.
> The R side does not seem to have something similar to:
> {{ def _should_silently_exclude(self, file_name):}}
> {{ return (file_name.endswith('.crc') or # Checksums}}
> {{ file_name.endswith('_$folder$') or # HDFS directories in S3}}
> {{ file_name.startswith('.') or # Hidden files starting with .}}
> {{ file_name.startswith('_') or # Hidden files starting with _}}
> {{ file_name in EXCLUDED_PARQUET_PATHS)}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)