You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Hari Shreedharan <hs...@cloudera.com> on 2013/07/01 23:47:33 UTC
Re: Review Request 11583: HDFS Sink should check if file is closed and retry
if it is not.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11583/#review22626
-----------------------------------------------------------
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
<https://reviews.apache.org/r/11583/#comment46313>
This parameter comes from the Flume configuration, not the HDFS configuration. So this will always be 30000. You probably need to pass it through a method or the configure method.
Am I missing something here?
- Hari Shreedharan
On June 28, 2013, 2:29 p.m., Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11583/
> -----------------------------------------------------------
>
> (Updated June 28, 2013, 2:29 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: 2007
> https://issues.apache.org/jira/browse/2007
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> We can use the new API added in HDFS-4525. We will need to use reflection though, so we can run against a version of HDFS which does not have this API.
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java bc3b383
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 2c2be6a
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java b8214be
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 0383744
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystemCloseRetryWrapper.java PRE-CREATION
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStreamCloseRetryWrapper.java PRE-CREATION
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java ffbdde0
>
> Diff: https://reviews.apache.org/r/11583/diff/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> Patch-0
> https://reviews.apache.org/media/uploaded/files/2013/05/31/FLUME-2007-0.patch
>
>
> Thanks,
>
> Ted Malaska
>
>
Re: Review Request 11583: HDFS Sink should check if file is closed and retry
if it is not.
Posted by Ted Malaska <te...@cloudera.com>.
> On July 1, 2013, 9:47 p.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java, lines 106-119
> > <https://reviews.apache.org/r/11583/diff/3/?file=312899#file312899line106>
> >
> > This parameter comes from the Flume configuration, not the HDFS configuration. So this will always be 30000. You probably need to pass it through a method or the configure method.
> >
> > Am I missing something here?
My bad. I thought it was a HDFS parameter. I will change it to a Flume config. That's easier anyways.
Thanks again. I will fix it tomorrow morning.
- Ted
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11583/#review22626
-----------------------------------------------------------
On June 28, 2013, 2:29 p.m., Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11583/
> -----------------------------------------------------------
>
> (Updated June 28, 2013, 2:29 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: 2007
> https://issues.apache.org/jira/browse/2007
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> We can use the new API added in HDFS-4525. We will need to use reflection though, so we can run against a version of HDFS which does not have this API.
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java bc3b383
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 2c2be6a
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java b8214be
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 0383744
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystemCloseRetryWrapper.java PRE-CREATION
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStreamCloseRetryWrapper.java PRE-CREATION
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java ffbdde0
>
> Diff: https://reviews.apache.org/r/11583/diff/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> Patch-0
> https://reviews.apache.org/media/uploaded/files/2013/05/31/FLUME-2007-0.patch
>
>
> Thanks,
>
> Ted Malaska
>
>