[jira] [Resolved] (METAMODEL-163) Composite/directory Resource for local files and HDFS files

["Kasper Sørensen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/METAMODEL-163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kasper Sørensen resolved METAMODEL-163.
---------------------------------------
       Resolution: Fixed
         Assignee: Dennis Du Krøger
    Fix Version/s: 4.4.0

This was fixed with the mentioned PR

> Composite/directory Resource for local files and HDFS files
> -----------------------------------------------------------
>
>                 Key: METAMODEL-163
>                 URL: https://issues.apache.org/jira/browse/METAMODEL-163
>             Project: Apache MetaModel
>          Issue Type: Improvement
>            Reporter: Kasper Sørensen
>            Assignee: Dennis Du Krøger
>             Fix For: 4.4.0
>
>
> A more and more common pattern in representing data is to have a directory with files of the same format which can be appended together to form a complete dataset. I see this especially in Hadoop scenarios where reducers as well as spark usually will create such "part" files in a directory and treat that directory almost as a logical file.
> I don't know if we can generalize this or if we need two separate implementations. But at least I would love to have a Resource implementation like this: Given a (local or HDFS) path that points to a directory, or maybe also to a wildcard-enabled expression, I would want to have a single Resource object that represents all the corresponding files in that directory/pattern.
> This would not only provide us with better interoperability with Hadoop result data, but it will also actually solve a long-standing request (in our company at least) to support multiple CSV files in one logical CsvDataContext.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)