You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Mike Percy <mp...@gmail.com> on 2012/05/21 09:24:03 UTC
Review Request: FLUME-1219: Fix race conditions in BucketWriter /
HDFSEventSink
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5175/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
BucketWriter refactoring: append() does all the work of open/close/roll. open() is a private method that takes no arguments. No abort() call. Only one constructor.
Far fewer entry points and code paths. I believe I've closed all or many of the race conditions and clarified the API responsibilities/semantics.
This addresses bug FLUME-1219.
https://issues.apache.org/jira/browse/FLUME-1219
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java d272f74
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 91cb822
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 397138b
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 0f78f37
Diff: https://reviews.apache.org/r/5175/diff
Testing
-------
Looks good under load. Unit tests pass.
Thanks,
Mike
Re: Review Request: FLUME-1219: Fix race conditions in BucketWriter /
HDFSEventSink
Posted by Mike Percy <mp...@gmail.com>.
> On 2012-05-21 08:09:28, Arvind Prabhakar wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java, line 40
> > <https://reviews.apache.org/r/5175/diff/1/?file=109725#file109725line40>
> >
> > I think to be on the safer side we should make this class all synchronized. It does introduce a potential for dead locks but without making this change, it is hard to analyze where other problems may be.
synchronized all public methods that can modify state
> On 2012-05-21 08:09:28, Arvind Prabhakar wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java, lines 488-503
> > <https://reviews.apache.org/r/5175/diff/1/?file=109726#file109726line488>
> >
> > This block is being done outside of a committed transaction. As a result, a failure in this will likely cause data loss. I suggest move it inside with a nested try/catch if necessary.
done
> On 2012-05-21 08:09:28, Arvind Prabhakar wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java, line 68
> > <https://reviews.apache.org/r/5175/diff/1/?file=109725#file109725line68>
> >
> > should be volatile or guarded
done, as well as a few other immutable or primitive variables readable via public methods
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5175/#review8005
-----------------------------------------------------------
On 2012-05-21 09:44:35, Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5175/
> -----------------------------------------------------------
>
> (Updated 2012-05-21 09:44:35)
>
>
> Review request for Flume.
>
>
> Summary
> -------
>
> BucketWriter refactoring: append() does all the work of open/close/roll. open() is a private method that takes no arguments. No abort() call. Only one constructor.
> Far fewer entry points and code paths. I believe I've closed all or many of the race conditions and clarified the API responsibilities/semantics.
>
>
> This addresses bug FLUME-1219.
> https://issues.apache.org/jira/browse/FLUME-1219
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 91cb822
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java d272f74
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 397138b
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 0f78f37
>
> Diff: https://reviews.apache.org/r/5175/diff
>
>
> Testing
> -------
>
> Looks good under load. Unit tests pass.
>
>
> Thanks,
>
> Mike
>
>
Re: Review Request: FLUME-1219: Fix race conditions in BucketWriter /
HDFSEventSink
Posted by Arvind Prabhakar <ar...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5175/#review8005
-----------------------------------------------------------
Thanks for doing this change Mike. Its long due technical debt. I suggest that once we review it at a high level, we can push this in and then root out more issues via actual testing.
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
<https://reviews.apache.org/r/5175/#comment17372>
I think to be on the safer side we should make this class all synchronized. It does introduce a potential for dead locks but without making this change, it is hard to analyze where other problems may be.
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
<https://reviews.apache.org/r/5175/#comment17370>
should be volatile or guarded
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
<https://reviews.apache.org/r/5175/#comment17371>
This block is being done outside of a committed transaction. As a result, a failure in this will likely cause data loss. I suggest move it inside with a nested try/catch if necessary.
- Arvind
On 2012-05-21 07:24:02, Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5175/
> -----------------------------------------------------------
>
> (Updated 2012-05-21 07:24:02)
>
>
> Review request for Flume.
>
>
> Summary
> -------
>
> BucketWriter refactoring: append() does all the work of open/close/roll. open() is a private method that takes no arguments. No abort() call. Only one constructor.
> Far fewer entry points and code paths. I believe I've closed all or many of the race conditions and clarified the API responsibilities/semantics.
>
>
> This addresses bug FLUME-1219.
> https://issues.apache.org/jira/browse/FLUME-1219
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java d272f74
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 91cb822
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 397138b
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 0f78f37
>
> Diff: https://reviews.apache.org/r/5175/diff
>
>
> Testing
> -------
>
> Looks good under load. Unit tests pass.
>
>
> Thanks,
>
> Mike
>
>
Re: Review Request: FLUME-1219: Fix race conditions in BucketWriter /
HDFSEventSink
Posted by Arvind Prabhakar <ar...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5175/#review8013
-----------------------------------------------------------
Ship it!
+1
Changes look good. Will commit if tests pass.
- Arvind
On 2012-05-21 10:24:48, Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5175/
> -----------------------------------------------------------
>
> (Updated 2012-05-21 10:24:48)
>
>
> Review request for Flume.
>
>
> Summary
> -------
>
> BucketWriter refactoring: append() does all the work of open/close/roll. open() is a private method that takes no arguments. No abort() call. Only one constructor.
> Far fewer entry points and code paths. I believe I've closed all or many of the race conditions and clarified the API responsibilities/semantics.
>
>
> This addresses bug FLUME-1219.
> https://issues.apache.org/jira/browse/FLUME-1219
>
>
> Diffs
> -----
>
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 91cb822
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java d272f74
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 397138b
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 0f78f37
>
> Diff: https://reviews.apache.org/r/5175/diff
>
>
> Testing
> -------
>
> Looks good under load. Unit tests pass.
>
>
> Thanks,
>
> Mike
>
>
Re: Review Request: FLUME-1219: Fix race conditions in BucketWriter /
HDFSEventSink
Posted by Mike Percy <mp...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5175/
-----------------------------------------------------------
(Updated 2012-05-21 10:24:48.919754)
Review request for Flume.
Changes
-------
Oops, triggered a timing-sensitive unit test by increasing the default call timeout.
Fixed the test. All unit tests pass.
Also, verified that this works fine on both a non-kerberized and a kerberized cluster.
Summary
-------
BucketWriter refactoring: append() does all the work of open/close/roll. open() is a private method that takes no arguments. No abort() call. Only one constructor.
Far fewer entry points and code paths. I believe I've closed all or many of the race conditions and clarified the API responsibilities/semantics.
This addresses bug FLUME-1219.
https://issues.apache.org/jira/browse/FLUME-1219
Diffs (updated)
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 91cb822
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java d272f74
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 397138b
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 0f78f37
Diff: https://reviews.apache.org/r/5175/diff
Testing
-------
Looks good under load. Unit tests pass.
Thanks,
Mike
Re: Review Request: FLUME-1219: Fix race conditions in BucketWriter /
HDFSEventSink
Posted by Mike Percy <mp...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5175/
-----------------------------------------------------------
(Updated 2012-05-21 09:44:35.214547)
Review request for Flume.
Changes
-------
Arvind, thanks for your insightful review. After the latest patch, the Kerberos-related thread safety issues appear to be gone, and the logs are as clean as the driven snow.
Changes in this patch:
Incorporated review feedback:
- volatile variables where needed
- Call BucketWriter.flush() before Transaction.commit() as appropriate
- synchronize the public methods of BucketWriter
Additional improvements:
- Unify configuration timeouts to just callTimeout (appendTimeout doesn't mean anything anymore)
- Increase default callTimeout to 10 seconds
- Acquire lock on getLoginUser() before calling Path.getFileSystem()
- this is because that call is not thread-safe under Kerberos
- in a non-kerberized cluster, this will just be the Unix login credential
Summary
-------
BucketWriter refactoring: append() does all the work of open/close/roll. open() is a private method that takes no arguments. No abort() call. Only one constructor.
Far fewer entry points and code paths. I believe I've closed all or many of the race conditions and clarified the API responsibilities/semantics.
This addresses bug FLUME-1219.
https://issues.apache.org/jira/browse/FLUME-1219
Diffs (updated)
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 91cb822
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java d272f74
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 397138b
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 0f78f37
Diff: https://reviews.apache.org/r/5175/diff
Testing
-------
Looks good under load. Unit tests pass.
Thanks,
Mike