You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Hari Shreedharan <hs...@cloudera.com> on 2013/06/07 20:09:04 UTC
Re: Review Request: HDFS Sink should check if file is closed and retry if it
is not.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11583/#review21588
-----------------------------------------------------------
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
<https://reviews.apache.org/r/11583/#comment44639>
Why are you using wait() here? A sleep should be enough. I don't think this needs to be in a synchronized block either - would more than 1 thread try to close the same file?
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
<https://reviews.apache.org/r/11583/#comment44640>
The loop already checks that you try invoking close only a fixed number of times. Also, currently we don't throw an exception if we could not close the file - I guess we should just keep the same semantics - catch it and log.
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
<https://reviews.apache.org/r/11583/#comment44641>
Is this new method really required? Does not look like it is.
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java
<https://reviews.apache.org/r/11583/#comment44642>
I don't see this class anywhere in the patch, maybe missed a git add?
- Hari Shreedharan
On May 31, 2013, 7:26 p.m., Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11583/
> -----------------------------------------------------------
>
> (Updated May 31, 2013, 7:26 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> We can use the new API added in HDFS-4525. We will need to use reflection though, so we can run against a version of HDFS which does not have this API.
>
>
> This addresses bug 2007.
> https://issues.apache.org/jira/browse/2007
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java bc3b383
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 2c2be6a
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java b8214be
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 0383744
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java ffbdde0
>
> Diff: https://reviews.apache.org/r/11583/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ted Malaska
>
>
Re: Review Request: HDFS Sink should check if file is closed and retry if it
is not.
Posted by Ted Malaska <te...@cloudera.com>.
> On June 7, 2013, 6:09 p.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java, lines 78-83
> > <https://reviews.apache.org/r/11583/diff/1/?file=299649#file299649line78>
> >
> > Is this new method really required? Does not look like it is.
It is required for unit testing. I forgot to add the unit testing file. Please review again after I add it in the next patch.
> On June 7, 2013, 6:09 p.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java, line 243
> > <https://reviews.apache.org/r/11583/diff/1/?file=299646#file299646line243>
> >
> > Why are you using wait() here? A sleep should be enough. I don't think this needs to be in a synchronized block either - would more than 1 thread try to close the same file?
Yes good point. I will use Thread.sleep() much cleaner
> On June 7, 2013, 6:09 p.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java, lines 252-255
> > <https://reviews.apache.org/r/11583/diff/1/?file=299646#file299646line252>
> >
> > The loop already checks that you try invoking close only a fixed number of times. Also, currently we don't throw an exception if we could not close the file - I guess we should just keep the same semantics - catch it and log.
Totally agreed
> On June 7, 2013, 6:09 p.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java, line 144
> > <https://reviews.apache.org/r/11583/diff/1/?file=299650#file299650line144>
> >
> > I don't see this class anywhere in the patch, maybe missed a git add?
Sorry rookie mistake on my part.
- Ted
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11583/#review21588
-----------------------------------------------------------
On May 31, 2013, 7:26 p.m., Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11583/
> -----------------------------------------------------------
>
> (Updated May 31, 2013, 7:26 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> We can use the new API added in HDFS-4525. We will need to use reflection though, so we can run against a version of HDFS which does not have this API.
>
>
> This addresses bug 2007.
> https://issues.apache.org/jira/browse/2007
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java bc3b383
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 2c2be6a
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java b8214be
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 0383744
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java ffbdde0
>
> Diff: https://reviews.apache.org/r/11583/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ted Malaska
>
>