[jira] [Created] (METAMODEL-163) Composite/directory Resource for local files and HDFS files

["Kasper Sørensen (JIRA)" <ji...@apache.org>]

Kasper Sørensen created METAMODEL-163:
-----------------------------------------

             Summary: Composite/directory Resource for local files and HDFS files
                 Key: METAMODEL-163
                 URL: https://issues.apache.org/jira/browse/METAMODEL-163
             Project: Apache MetaModel
          Issue Type: Improvement
            Reporter: Kasper Sørensen


A more and more common pattern in representing data is to have a directory with files of the same format which can be appended together to form a complete dataset. I see this especially in Hadoop scenarios where reducers as well as spark usually will create such "part" files in a directory and treat that directory almost as a logical file.

I don't know if we can generalize this or if we need two separate implementations. But at least I would love to have a Resource implementation like this: Given a (local or HDFS) path that points to a directory, or maybe also to a wildcard-enabled expression, I would want to have a single Resource object that represents all the corresponding files in that directory/pattern.

This would not only provide us with better interoperability with Hadoop result data, but it will also actually solve a long-standing request (in our company at least) to support multiple CSV files in one logical CsvDataContext.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)