You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Mingjie Lai (JIRA)" <ji...@apache.org> on 2012/05/17 02:50:08 UTC
[jira] [Created] (FLUME-1209) Support file name extension for HDFS
sink
Mingjie Lai created FLUME-1209:
----------------------------------
Summary: Support file name extension for HDFS sink
Key: FLUME-1209
URL: https://issues.apache.org/jira/browse/FLUME-1209
Project: Flume
Issue Type: Improvement
Components: Sinks+Sources
Affects Versions: NG alpha 1
Reporter: Mingjie Lai
Assignee: Mingjie Lai
Here is a common use case for using pig to analyze hdfs files collected by flume:
{code}
data = LOAD '/data/mydata/*' org.apache.pig.piggybank.storage.avro.AvroStorage();
{code}
Since the current hdfs sink creates tmp files under the current directory, pig will complain loading file failure.
At Flume-965, Brock mentioned we can add a filePostfix attribute to hdfs sink. I think it's a good way to solve the issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1209) Support file name extension for
HDFS sink
Posted by "Mingjie Lai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280326#comment-13280326 ]
Mingjie Lai commented on FLUME-1209:
------------------------------------
Pls help to review: https://reviews.apache.org/r/5179/
(I was told people don't want to see reviewboard notifications at jira, so I didn't put the branch number there. )
> Support file name extension for HDFS sink
> -----------------------------------------
>
> Key: FLUME-1209
> URL: https://issues.apache.org/jira/browse/FLUME-1209
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Mingjie Lai
> Assignee: Mingjie Lai
> Fix For: v1.2.0
>
>
> Here is a common use case for using pig to analyze hdfs files collected by flume:
> {code}
> data = LOAD '/data/mydata/*' org.apache.pig.piggybank.storage.avro.AvroStorage();
> {code}
> Since the current hdfs sink creates tmp files under the current directory, pig will complain loading file failure.
> At Flume-965, Brock mentioned we can add a filePostfix attribute to hdfs sink. I think it's a good way to solve the issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1209) Support file name extension for HDFS
sink
Posted by "Mingjie Lai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mingjie Lai updated FLUME-1209:
-------------------------------
Affects Version/s: (was: NG alpha 1)
v1.1.0
Fix Version/s: v1.2.0
> Support file name extension for HDFS sink
> -----------------------------------------
>
> Key: FLUME-1209
> URL: https://issues.apache.org/jira/browse/FLUME-1209
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Mingjie Lai
> Assignee: Mingjie Lai
> Fix For: v1.2.0
>
>
> Here is a common use case for using pig to analyze hdfs files collected by flume:
> {code}
> data = LOAD '/data/mydata/*' org.apache.pig.piggybank.storage.avro.AvroStorage();
> {code}
> Since the current hdfs sink creates tmp files under the current directory, pig will complain loading file failure.
> At Flume-965, Brock mentioned we can add a filePostfix attribute to hdfs sink. I think it's a good way to solve the issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira