You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Mingjie Lai <mj...@gmail.com> on 2011/10/20 08:54:48 UTC
Regarding recent InterruptedException issues.
Hi.
As some ones on the list, I also noticed the InterruptedException issue
of DirectDriver, but under different situation. If I reconfigure a node,
the node always raises some InterruptedException in log. It seems the
node's source.close() not getting called at all. It can explain others'
observation, such as InterruptedExc of rollsink and colletorSink, since
they all keep closing source/sink once a while. If a Driver cannot see
the thread exiting gracefully, it will force the thread to exit by
interrupt, which can bring unexpected behavior for different source/sinks.
It might be brought by a patch of Flume-596 which is a big refactor of
Driver. I rolled the source code back to
036a5106a0b15e25029e458349d9e6bf2142ea21, but cannot reproduce the
issue. Then I can see it again if cherry-picked Flume-596.
I opened https://issues.apache.org/jira/browse/FLUME-808 for detail
discussion.
Thanks,
Mingjie
Re: Regarding recent InterruptedException issues.
Posted by Cameron Gandevia <cg...@gmail.com>.
Interesting. I will rollback to the same revision and test with my
configuration. Thanks for the heads up.
On Wed, Oct 19, 2011 at 11:54 PM, Mingjie Lai <mj...@gmail.com> wrote:
>
> Hi.
>
> As some ones on the list, I also noticed the InterruptedException issue of
> DirectDriver, but under different situation. If I reconfigure a node, the
> node always raises some InterruptedException in log. It seems the node's
> source.close() not getting called at all. It can explain others'
> observation, such as InterruptedExc of rollsink and colletorSink, since they
> all keep closing source/sink once a while. If a Driver cannot see the thread
> exiting gracefully, it will force the thread to exit by interrupt, which can
> bring unexpected behavior for different source/sinks.
>
> It might be brought by a patch of Flume-596 which is a big refactor of
> Driver. I rolled the source code back to 036a5106a0b15e25029e458349d9e6**bf2142ea21,
> but cannot reproduce the issue. Then I can see it again if cherry-picked
> Flume-596.
>
> I opened https://issues.apache.org/**jira/browse/FLUME-808<https://issues.apache.org/jira/browse/FLUME-808>for detail discussion.
>
> Thanks,
> Mingjie
>
--
Thanks
Cameron Gandevia
Re: Regarding recent InterruptedException issues.
Posted by Cameron Gandevia <cg...@gmail.com>.
Sweet saw your patch I will give it a test run in our environment.
On Thu, Oct 20, 2011 at 12:01 PM, Mingjie Lai <mj...@gmail.com> wrote:
> Sorry. When writing the email, I was using the old cloudera git.
>
> For trunk, you can git-log, and search keyword for FLUME-596.
>
> -----
> commit 9b726222422d5f415bbb737e8636e4**4cb1a0ec95
> Author: Andrew Bayer <ab...@apache.org>
> Date: Tue Aug 2 16:05:58 2011 +0000
>
> FLUME-596: Simplify driver by removing unneeded DriverListener
>
> From: Jonathan Hsieh <jo...@cloudera.com>
>
> git-svn-id: https://svn.apache.org/repos/**asf/incubator/flume/trunk@**
> 1155903 <ht...@1155903>13f79535-47bb-031
>
> commit feb66308e00e116aae2ad21a712f00**2c3cefec8d
> Author: Andrew Bayer <ab...@apache.org>
> Date: Tue Aug 2 16:05:57 2011 +0000
>
> FLUME-593: Node should should only spawn logical node specified on
> master
> - Previously, node would always spawn a null|null node and spawn a
> separate thread to reconf
>
> From: Jonathan Hsieh <jo...@cloudera.com>
>
> git-svn-id: https://svn.apache.org/repos/**asf/incubator/flume/trunk@**
> 1155902 <ht...@1155902>13f79535-47bb-031
>
> -----
>
>
>
> On 10/20/2011 11:39 AM, Cameron Gandevia wrote:
>
>> I can't seem to find the commit you rolled back to.
>>
>> git branch --contains 036a5106a0b15e25029e458349d9e6**bf2142ea21
>> error: no such commit 036a5106a0b15e25029e458349d9e6**bf2142ea21
>>
>> I am going to try 0948ca70471b6bdd147fd6c74c9f1d**59bd59369c
>>
>> On Wed, Oct 19, 2011 at 11:54 PM, Mingjie Lai <mjlai09@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>>
>> Hi.
>>
>> As some ones on the list, I also noticed the InterruptedException
>> issue of DirectDriver, but under different situation. If I
>> reconfigure a node, the node always raises some InterruptedException
>> in log. It seems the node's source.close() not getting called at
>> all. It can explain others' observation, such as InterruptedExc of
>> rollsink and colletorSink, since they all keep closing source/sink
>> once a while. If a Driver cannot see the thread exiting gracefully,
>> it will force the thread to exit by interrupt, which can bring
>> unexpected behavior for different source/sinks.
>>
>> It might be brought by a patch of Flume-596 which is a big refactor
>> of Driver. I rolled the source code back to
>> 036a5106a0b15e25029e458349d9e6**__bf2142ea21, but cannot reproduce the
>>
>> issue. Then I can see it again if cherry-picked Flume-596.
>>
>> I opened https://issues.apache.org/__**jira/browse/FLUME-808<https://issues.apache.org/__jira/browse/FLUME-808>
>>
>> <https://issues.apache.org/**jira/browse/FLUME-808<https://issues.apache.org/jira/browse/FLUME-808>>
>> for detail discussion.
>>
>> Thanks,
>> Mingjie
>>
>>
>>
>>
>> --
>> Thanks
>>
>> Cameron Gandevia
>>
>
--
Thanks
Cameron Gandevia
Re: Regarding recent InterruptedException issues.
Posted by Mingjie Lai <mj...@gmail.com>.
Sorry. When writing the email, I was using the old cloudera git.
For trunk, you can git-log, and search keyword for FLUME-596.
-----
commit 9b726222422d5f415bbb737e8636e44cb1a0ec95
Author: Andrew Bayer <ab...@apache.org>
Date: Tue Aug 2 16:05:58 2011 +0000
FLUME-596: Simplify driver by removing unneeded DriverListener
From: Jonathan Hsieh <jo...@cloudera.com>
git-svn-id:
https://svn.apache.org/repos/asf/incubator/flume/trunk@1155903
13f79535-47bb-031
commit feb66308e00e116aae2ad21a712f002c3cefec8d
Author: Andrew Bayer <ab...@apache.org>
Date: Tue Aug 2 16:05:57 2011 +0000
FLUME-593: Node should should only spawn logical node specified on
master
- Previously, node would always spawn a null|null node and spawn a
separate thread to reconf
From: Jonathan Hsieh <jo...@cloudera.com>
git-svn-id:
https://svn.apache.org/repos/asf/incubator/flume/trunk@1155902
13f79535-47bb-031
-----
On 10/20/2011 11:39 AM, Cameron Gandevia wrote:
> I can't seem to find the commit you rolled back to.
>
> git branch --contains 036a5106a0b15e25029e458349d9e6bf2142ea21
> error: no such commit 036a5106a0b15e25029e458349d9e6bf2142ea21
>
> I am going to try 0948ca70471b6bdd147fd6c74c9f1d59bd59369c
>
> On Wed, Oct 19, 2011 at 11:54 PM, Mingjie Lai <mjlai09@gmail.com
> <ma...@gmail.com>> wrote:
>
>
> Hi.
>
> As some ones on the list, I also noticed the InterruptedException
> issue of DirectDriver, but under different situation. If I
> reconfigure a node, the node always raises some InterruptedException
> in log. It seems the node's source.close() not getting called at
> all. It can explain others' observation, such as InterruptedExc of
> rollsink and colletorSink, since they all keep closing source/sink
> once a while. If a Driver cannot see the thread exiting gracefully,
> it will force the thread to exit by interrupt, which can bring
> unexpected behavior for different source/sinks.
>
> It might be brought by a patch of Flume-596 which is a big refactor
> of Driver. I rolled the source code back to
> 036a5106a0b15e25029e458349d9e6__bf2142ea21, but cannot reproduce the
> issue. Then I can see it again if cherry-picked Flume-596.
>
> I opened https://issues.apache.org/__jira/browse/FLUME-808
> <https://issues.apache.org/jira/browse/FLUME-808> for detail discussion.
>
> Thanks,
> Mingjie
>
>
>
>
> --
> Thanks
>
> Cameron Gandevia
Re: Regarding recent InterruptedException issues.
Posted by Cameron Gandevia <cg...@gmail.com>.
I can't seem to find the commit you rolled back to.
git branch --contains 036a5106a0b15e25029e458349d9e6bf2142ea21
error: no such commit 036a5106a0b15e25029e458349d9e6bf2142ea21
I am going to try 0948ca70471b6bdd147fd6c74c9f1d59bd59369c
On Wed, Oct 19, 2011 at 11:54 PM, Mingjie Lai <mj...@gmail.com> wrote:
>
> Hi.
>
> As some ones on the list, I also noticed the InterruptedException issue of
> DirectDriver, but under different situation. If I reconfigure a node, the
> node always raises some InterruptedException in log. It seems the node's
> source.close() not getting called at all. It can explain others'
> observation, such as InterruptedExc of rollsink and colletorSink, since they
> all keep closing source/sink once a while. If a Driver cannot see the thread
> exiting gracefully, it will force the thread to exit by interrupt, which can
> bring unexpected behavior for different source/sinks.
>
> It might be brought by a patch of Flume-596 which is a big refactor of
> Driver. I rolled the source code back to 036a5106a0b15e25029e458349d9e6**bf2142ea21,
> but cannot reproduce the issue. Then I can see it again if cherry-picked
> Flume-596.
>
> I opened https://issues.apache.org/**jira/browse/FLUME-808<https://issues.apache.org/jira/browse/FLUME-808>for detail discussion.
>
> Thanks,
> Mingjie
>
--
Thanks
Cameron Gandevia