You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Hari Shreedharan <hs...@cloudera.com> on 2013/12/20 09:17:55 UTC
Review Request 16416: FLUME-2265. Closed bucket writers should be removed
from sfwriters map
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16416/
-----------------------------------------------------------
Review request for Flume.
Bugs: FLUME-2265
https://issues.apache.org/jira/browse/FLUME-2265
Repository: flume-git
Description
-------
On idle or roll time based close, the bucket writer is closed and the bucket writer is removed from sfWriters map.
The patch introduces a new exception, which is thrown when an append tries to write to a bucket writer that is closed based on time (roll or idle).
The sink then catches this exception, creates a new bucket writer and writes to that one.
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketClosedException.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 200d457
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java f0a6e4b
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 63ab5af
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadWriterFactory.java f5d0808
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestSeqWriter.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestWriterFactory.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java f741e03
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5b7cec9
pom.xml 267925f
Diff: https://reviews.apache.org/r/16416/diff/
Testing
-------
Thanks,
Hari Shreedharan
Re: Review Request 16416: FLUME-2265. Closed bucket writers should be
removed from sfwriters map
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16416/#review30749
-----------------------------------------------------------
Good job Hari, couple of notes:
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
<https://reviews.apache.org/r/16416/#comment58854>
Nit: This method do not appear to be used.
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestWriterFactory.java
<https://reviews.apache.org/r/16416/#comment58853>
Nit: Unused variable.
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
<https://reviews.apache.org/r/16416/#comment58856>
Super critical: Use your own message!
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
<https://reviews.apache.org/r/16416/#comment58855>
Nit: Unused variable.
pom.xml
<https://reviews.apache.org/r/16416/#comment58852>
It seems that we already do have this dependency in pom.xml making this patch fail to apply:
https://github.com/apache/flume/blob/trunk/pom.xml#L777
It seems to be added by FLUME-2155.
Jarcec
- Jarek Cecho
On Dec. 20, 2013, 8:18 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16416/
> -----------------------------------------------------------
>
> (Updated Dec. 20, 2013, 8:18 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2265
> https://issues.apache.org/jira/browse/FLUME-2265
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> On idle or roll time based close, the bucket writer is closed and the bucket writer is removed from sfWriters map.
> The patch introduces a new exception, which is thrown when an append tries to write to a bucket writer that is closed based on time (roll or idle).
> The sink then catches this exception, creates a new bucket writer and writes to that one.
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketClosedException.java PRE-CREATION
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 200d457
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java f0a6e4b
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 63ab5af
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadWriterFactory.java f5d0808
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestSeqWriter.java PRE-CREATION
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestWriterFactory.java PRE-CREATION
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java f741e03
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5b7cec9
> pom.xml 267925f
>
> Diff: https://reviews.apache.org/r/16416/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request 16416: FLUME-2265. Closed bucket writers should be
removed from sfwriters map
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16416/#review30820
-----------------------------------------------------------
Ship it!
Ship It!
- Jarek Cecho
On Dec. 20, 2013, 5:58 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16416/
> -----------------------------------------------------------
>
> (Updated Dec. 20, 2013, 5:58 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2265
> https://issues.apache.org/jira/browse/FLUME-2265
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> On idle or roll time based close, the bucket writer is closed and the bucket writer is removed from sfWriters map.
> The patch introduces a new exception, which is thrown when an append tries to write to a bucket writer that is closed based on time (roll or idle).
> The sink then catches this exception, creates a new bucket writer and writes to that one.
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketClosedException.java PRE-CREATION
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 200d457
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java f0a6e4b
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 63ab5af
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadWriterFactory.java f5d0808
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestSeqWriter.java PRE-CREATION
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestWriterFactory.java PRE-CREATION
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java f741e03
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5b7cec9
>
> Diff: https://reviews.apache.org/r/16416/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request 16416: FLUME-2265. Closed bucket writers should be
removed from sfwriters map
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16416/
-----------------------------------------------------------
(Updated Dec. 20, 2013, 5:58 p.m.)
Review request for Flume.
Changes
-------
Update based on Jarcec's comments
Bugs: FLUME-2265
https://issues.apache.org/jira/browse/FLUME-2265
Repository: flume-git
Description
-------
On idle or roll time based close, the bucket writer is closed and the bucket writer is removed from sfWriters map.
The patch introduces a new exception, which is thrown when an append tries to write to a bucket writer that is closed based on time (roll or idle).
The sink then catches this exception, creates a new bucket writer and writes to that one.
Diffs (updated)
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketClosedException.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 200d457
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java f0a6e4b
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 63ab5af
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadWriterFactory.java f5d0808
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestSeqWriter.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestWriterFactory.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java f741e03
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5b7cec9
Diff: https://reviews.apache.org/r/16416/diff/
Testing
-------
Thanks,
Hari Shreedharan
Re: Review Request 16416: FLUME-2265. Closed bucket writers should be
removed from sfwriters map
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16416/
-----------------------------------------------------------
(Updated Dec. 20, 2013, 8:18 a.m.)
Review request for Flume.
Bugs: FLUME-2265
https://issues.apache.org/jira/browse/FLUME-2265
Repository: flume-git
Description
-------
On idle or roll time based close, the bucket writer is closed and the bucket writer is removed from sfWriters map.
The patch introduces a new exception, which is thrown when an append tries to write to a bucket writer that is closed based on time (roll or idle).
The sink then catches this exception, creates a new bucket writer and writes to that one.
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketClosedException.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 200d457
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java f0a6e4b
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 63ab5af
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadWriterFactory.java f5d0808
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestSeqWriter.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestWriterFactory.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java f741e03
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5b7cec9
pom.xml 267925f
Diff: https://reviews.apache.org/r/16416/diff/
Testing
-------
Thanks,
Hari Shreedharan