You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flume.apache.org by Balázs Donát Bessenyei <be...@cloudera.com> on 2016/09/19 12:50:27 UTC

Re: Review Request 51204: FLUME-2973: Deadlock in hdfs sink

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51204/#review149459
-----------------------------------------------------------



If you could please create a unit-test of the tool that you wrote, that would be very useful.


flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java (lines 465 - 472)
<https://reviews.apache.org/r/51204/#comment217105>

    Is there no way to simplify this code?



flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java (lines 80 - 104)
<https://reviews.apache.org/r/51204/#comment217106>

    Can these calls be refactored to reduce copy-paste?


- Bal�zs Don�t Bessenyei


On Aug. 18, 2016, 12:56 p.m., Denes Arvay wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51204/
> -----------------------------------------------------------
> 
> (Updated Aug. 18, 2016, 12:56 p.m.)
> 
> 
> Review request for Flume, Bal�zs Don�t Bessenyei, Mike Percy, and Attila Simon.
> 
> 
> Bugs: FLUME-2973
>     https://issues.apache.org/jira/browse/FLUME-2973
> 
> 
> Repository: flume-git
> 
> 
> Description
> -------
> 
> Automatic close of BucketWriters (when open file count reached `hdfs.maxOpenFiles`) and the file rolling thread can end up in deadlock.
> 
> 
> Diffs
> -----
> 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java b096410 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 741f01e 
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 742deb0 
> 
> Diff: https://reviews.apache.org/r/51204/diff/
> 
> 
> Testing
> -------
> 
> `flume-hdfs-sink` tests pass. No new tests were added.
> I created this script to validate the solution: https://gist.github.com/adenes/96503a6e737f9604ab3ee9397a5809ff
> 
> 
> Thanks,
> 
> Denes Arvay
> 
>