You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Juhani Connolly (JIRA)" <ji...@apache.org> on 2012/05/31 05:58:23 UTC
[jira] [Created] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Juhani Connolly created FLUME-1246:
--------------------------------------
Summary: FileChannel failing to start, also shutdown impossible without kill
Key: FLUME-1246
URL: https://issues.apache.org/jira/browse/FLUME-1246
Project: Flume
Issue Type: Bug
Components: Channel
Affects Versions: v1.2.0
Environment: CentOS 5.4
Reporter: Juhani Connolly
Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
Config:
test.channels.ch1.type = file
test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
test.sources.top.type = exec
test.sources.top.command = /usr/bin top -b -d 1
test.sources.top.restart = true
test.sources.top.restartThrottle = 1000
test.sources.top.interceptors = ts
test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
test.sources.top.channels = ch1
test.sinks.log.type = logger
test.sinks.log.channel = ch1
test.channels = ch1
test.sources = top
test.sinks = log
attaching logs with general/lifecycle loglevel down to debug.
A solution to this is probably going to be just improving error reporting.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Created] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by Hari Shreedharan <hs...@cloudera.com>.
Hi Inder
This issue you are saying was fixed in FLUME-1236. The issue Juhani is
seeing is different and exists even after that fix.
On Thursday, May 31, 2012, Inder SIngh (JIRA) wrote:
>
> [
> https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Inder SIngh updated FLUME-1246:
> -------------------------------
>
> Attachment: flume.log
>
> Flume log capturing channel not starting exceptions
>
> > FileChannel failing to start, also shutdown impossible without kill
> > --------------------------------------------------------------------
> >
> > Key: FLUME-1246
> > URL: https://issues.apache.org/jira/browse/FLUME-1246
> > Project: Flume
> > Issue Type: Bug
> > Components: Channel
> > Affects Versions: v1.2.0
> > Environment: CentOS 5.4
> > Reporter: Juhani Connolly
> > Attachments: flume.log, flume.log
> >
> >
> > Reduced to a minimal configuration for simplicity. I can recreate this
> on some machines, and not others. I wouldn't be surprised if it is some
> machines specific issue(test machines on CentOS5.4. On some it worked,
> others not), however whatever exception was thrown when attempting to get
> created is consumed and never passed onwards
> > Config:
> > test.channels.ch1.type = file
> > test.channels.ch1.checkpointDir =
> /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> > test.channels.ch1.dataDirs =
> /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> > test.sources.top.type = exec
> > test.sources.top.command = /usr/bin top -b -d 1
> > test.sources.top.restart = true
> > test.sources.top.restartThrottle = 1000
> > test.sources.top.interceptors = ts
> > test.sources.top.interceptors.ts.type =
> org.apache.flume.interceptor.TimestampInterceptor$Builder
> > test.sources.top.channels = ch1
> > test.sinks.log.type = logger
> > test.sinks.log.channel = ch1
> > test.channels = ch1
> > test.sources = top
> > test.sinks = log
> > attaching logs with general/lifecycle loglevel down to debug.
> > A solution to this is probably going to be just improving error
> reporting.
> > Another possibly more important element is that flume enters a state
> from which it cannot shutdown without kill -9. It looks like the Interrupts
> are getting swallowed up silently somewhere
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>
[jira] [Updated] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Inder SIngh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Inder SIngh updated FLUME-1246:
-------------------------------
Attachment: flume.log
Flume log capturing channel not starting exceptions
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Will McQueen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286745#comment-13286745 ]
Will McQueen commented on FLUME-1246:
-------------------------------------
With a config similary to Juhani's, I'm now able to reproduce FLUME-1232 (using a custom timestamp interceptor, since I'm not synced to the latest bits).
agent.channels = c1
agent.sources = r1
agent.sinks = k1
agent.channels.c1.type = FILE
agent.channels.c1.checkpointDir = /tmp/flume-check
agent.channels.c1.dataDirs = /tmp/flume-data
agent.sources.r1.channels = c1
agent.sources.r1.type = EXEC
agent.sources.r1.command = /usr/bin/top -b -d 1
agent.sources.r1.restart = true
agent.sources.r1.restartThrottle = 1000
agent.sources.r1.interceptors = i1
agent.sources.r1.interceptors.i1.type = org.apache.flume.interceptor.custom.TimestampInterceptor$Builder
agent.sinks.k1.channel = c1
agent.sinks.k1.type = LOGGER
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-1246:
------------------------------------
Attachment: flume.log
Sorry about the large file, primarily due to SEQ source
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log, flume.log, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1246) FileChannel has a dependence on
hadoop IO and will without logs, making clean shutdown impossible
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Juhani Connolly updated FLUME-1246:
-----------------------------------
Description:
FileChannels Event implementation uses Hadoop IO's Writable for serialization. Without it a class loader error is thrown but never caught and logged killing the thread invisibly. This repeats and puts flume into a state where interrupts are not responded to, making clean shutdown impossible
old desc for posterity:
Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
Config:
test.channels.ch1.type = file
test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
test.sources.top.type = exec
test.sources.top.command = /usr/bin top -b -d 1
test.sources.top.restart = true
test.sources.top.restartThrottle = 1000
test.sources.top.interceptors = ts
test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
test.sources.top.channels = ch1
test.sinks.log.type = logger
test.sinks.log.channel = ch1
test.channels = ch1
test.sources = top
test.sinks = log
attaching logs with general/lifecycle loglevel down to debug.
A solution to this is probably going to be just improving error reporting.
Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
was:
Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
Config:
test.channels.ch1.type = file
test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
test.sources.top.type = exec
test.sources.top.command = /usr/bin top -b -d 1
test.sources.top.restart = true
test.sources.top.restartThrottle = 1000
test.sources.top.interceptors = ts
test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
test.sources.top.channels = ch1
test.sinks.log.type = logger
test.sinks.log.channel = ch1
test.channels = ch1
test.sources = top
test.sinks = log
attaching logs with general/lifecycle loglevel down to debug.
A solution to this is probably going to be just improving error reporting.
Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
Summary: FileChannel has a dependence on hadoop IO and will without logs, making clean shutdown impossible (was: FileChannel failing to start, also shutdown impossible without kill)
> FileChannel has a dependence on hadoop IO and will without logs, making clean shutdown impossible
> -------------------------------------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> FileChannels Event implementation uses Hadoop IO's Writable for serialization. Without it a class loader error is thrown but never caught and logged killing the thread invisibly. This repeats and puts flume into a state where interrupts are not responded to, making clean shutdown impossible
> old desc for posterity:
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287198#comment-13287198 ]
Hari Shreedharan commented on FLUME-1246:
-----------------------------------------
Agreed the channel creation does not take time. But channels are started only much later. This is the order of events:
Create new components
Stop all old components if they exist
Configure new components
Start new components.
>From your logs, it does not seem like the sinks have been created either. I cannot reproduce a situation where start is not being called on the channel, and the flow of the conf-poller thread is quite deterministic.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (FLUME-1246) FileChannel failing to start,
also shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287192#comment-13287192 ]
Juhani Connolly edited comment on FLUME-1246 at 6/1/12 5:47 AM:
----------------------------------------------------------------
top worked fine with ExecSource for me, I had it running through a MemoryChannel no problem.
Yesterdays issue also could only be reproduced on some systems so I'm not too surprised that this config works in some places.
However even with full debug output it's not providing any clues as to what is wrong(with either yesterday or todays builds/problems)
You can notice from my log there is 4 seconds between channel creation and the stop. It may not seem like long but it should be enough. I've repeated this with much longer pauses before stopping it, and can post more logs if needed.
edit: oops took the wrong part from your logs... The part with creating instance of filechannel appears to not appear in them, since logs are at INFO level
was (Author: juhanic):
top worked fine with ExecSource for me, I had it running through a MemoryChannel no problem.
Yesterdays issue also could only be reproduced on some systems so I'm not too surprised that this config works in some places.
However even with full debug output it's not providing any clues as to what is wrong(with either yesterday or todays builds/problems)
You can notice from my log there is 4 seconds between channel creation and the stop. It may not seem like long but it should be enough. I've repeated this with much longer pauses before stopping it, and can post more logs if needed.
edit: oops took the wrong part from your logs... The part with creating instance of filechannel appears to not appear in them, presumably because of the difference in cutoff point
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287143#comment-13287143 ]
Juhani Connolly commented on FLUME-1246:
----------------------------------------
That log is from yesterday before I pulled the new patches. The lack of a start is happening with the following conf(no need for the completely irrelevant timestamp for this test):
test.channels.ch1.type = file
test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
test.sources.top.type = exec
test.sources.top.command = /usr/bin/top -b -d 1
test.sources.top.restart = true
test.sources.top.restartThrottle = 1000
test.sources.top.channels = ch1
test.sinks.log.type = logger
test.sinks.log.channel = ch1
test.channels = ch1
test.sources = top
test.sinks = log
I'm also attaching a new log
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log, flume.log, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1246) FileChannel hangs silently when
Hadoop libs not found
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy updated FLUME-1246:
------------------------------
Summary: FileChannel hangs silently when Hadoop libs not found (was: FileChannel has a dependence on hadoop IO and will without logs, making clean shutdown impossible)
Changing summary to more accurately reflect latest intentions on ticket (AFAICT). Please change again if needed.
> FileChannel hangs silently when Hadoop libs not found
> -----------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Priority: Blocker
> Fix For: v1.2.0
>
> Attachments: FLUME-1246.patch, flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> FileChannels Event implementation uses Hadoop IO's Writable for serialization. Without it a class loader error is thrown but never caught and logged killing the thread invisibly. This repeats and puts flume into a state where interrupts are not responded to, making clean shutdown impossible
> old desc for posterity:
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286322#comment-13286322 ]
Hari Shreedharan commented on FLUME-1246:
-----------------------------------------
Ok. This seems to be some other issue. The channel is definitely not starting.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel hangs silently when
Hadoop libs not found
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393487#comment-13393487 ]
Mike Percy commented on FLUME-1246:
-----------------------------------
Created FLUME-1285 to track the Hadoop IO class dependency
> FileChannel hangs silently when Hadoop libs not found
> -----------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Priority: Blocker
> Fix For: v1.2.0
>
> Attachments: FLUME-1246.patch, flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> FileChannels Event implementation uses Hadoop IO's Writable for serialization. Without it a class loader error is thrown but never caught and logged killing the thread invisibly. This repeats and puts flume into a state where interrupts are not responded to, making clean shutdown impossible
> old desc for posterity:
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Juhani Connolly updated FLUME-1246:
-----------------------------------
Description:
Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
Config:
test.channels.ch1.type = file
test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
test.sources.top.type = exec
test.sources.top.command = /usr/bin top -b -d 1
test.sources.top.restart = true
test.sources.top.restartThrottle = 1000
test.sources.top.interceptors = ts
test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
test.sources.top.channels = ch1
test.sinks.log.type = logger
test.sinks.log.channel = ch1
test.channels = ch1
test.sources = top
test.sinks = log
attaching logs with general/lifecycle loglevel down to debug.
A solution to this is probably going to be just improving error reporting.
Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
was:
Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
Config:
test.channels.ch1.type = file
test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
test.sources.top.type = exec
test.sources.top.command = /usr/bin top -b -d 1
test.sources.top.restart = true
test.sources.top.restartThrottle = 1000
test.sources.top.interceptors = ts
test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
test.sources.top.channels = ch1
test.sinks.log.type = logger
test.sinks.log.channel = ch1
test.channels = ch1
test.sources = top
test.sinks = log
attaching logs with general/lifecycle loglevel down to debug.
A solution to this is probably going to be just improving error reporting.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287118#comment-13287118 ]
Hari Shreedharan commented on FLUME-1246:
-----------------------------------------
Juhani,
I am fairly sure start is being called. From the log you posted earlier today:
2012-05-31 12:39:42,512 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel ch1
and the earlier log too:
2012-05-31 12:51:15,100 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel ch1
This is from here:
for (Entry<String, Channel> entry :
nodeConfiguration.getChannels().entrySet()) {
try{
logger.info("Starting Channel " + entry.getKey());
nodeSupervisor.supervise(entry.getValue(),
new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
} catch (Exception e){
logger.error("Error while starting {}", entry.getValue(), e);
}
}
The channel failing to start does not seem related to start not getting called. If you try running it through a debugger and put a break point at start you can verify.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287105#comment-13287105 ]
Juhani Connolly commented on FLUME-1246:
----------------------------------------
looks like one of yesterdays patches broke things further... start() is not getting called(as evidenced by a lack of entry in logs). This is also happening on every server I test on.
On the positive side, it shuts down properly now(presumably because the monitoring loop is not started on it) :)
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-1246) FileChannel hangs silently when
Hadoop libs not found
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Juhani Connolly resolved FLUME-1246.
------------------------------------
Resolution: Fixed
Committed.
In the long term FileChannel and ResumableMemoryChannel should both be refactored to use a serialization method that doesn't add a hadoop io dependency
> FileChannel hangs silently when Hadoop libs not found
> -----------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Priority: Blocker
> Fix For: v1.2.0
>
> Attachments: FLUME-1246.patch, flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> FileChannels Event implementation uses Hadoop IO's Writable for serialization. Without it a class loader error is thrown but never caught and logged killing the thread invisibly. This repeats and puts flume into a state where interrupts are not responded to, making clean shutdown impossible
> old desc for posterity:
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-1246:
------------------------------------
Attachment: flume-hari-2012May312217PST.log
Removing the old file and renamed the same and uploaded
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (FLUME-1246) FileChannel failing to start,
also shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287143#comment-13287143 ]
Juhani Connolly edited comment on FLUME-1246 at 6/1/12 4:30 AM:
----------------------------------------------------------------
That log is from yesterday before I pulled the new patches. The lack of a start is happening with the following conf(no need for the completely irrelevant timestamp for this test):
test.channels.ch1.type = file
test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
test.sources.top.type = exec
test.sources.top.command = /usr/bin/top -b -d 1
test.sources.top.restart = true
test.sources.top.restartThrottle = 1000
test.sources.top.channels = ch1
test.sinks.log.type = logger
test.sinks.log.channel = ch1
test.channels = ch1
test.sources = top
test.sinks = log
I'm also attaching a new log
edit: the original problem is likely still there, it is just getting hidden by the fact that now for some reason the channel is not even having start() called(refer to flume.log20120601)
was (Author: juhanic):
That log is from yesterday before I pulled the new patches. The lack of a start is happening with the following conf(no need for the completely irrelevant timestamp for this test):
test.channels.ch1.type = file
test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
test.sources.top.type = exec
test.sources.top.command = /usr/bin/top -b -d 1
test.sources.top.restart = true
test.sources.top.restartThrottle = 1000
test.sources.top.channels = ch1
test.sinks.log.type = logger
test.sinks.log.channel = ch1
test.channels = ch1
test.sources = top
test.sinks = log
I'm also attaching a new log
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (FLUME-1246) FileChannel failing to start,
also shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286857#comment-13286857 ]
Hari Shreedharan edited comment on FLUME-1246 at 5/31/12 7:32 PM:
------------------------------------------------------------------
Inder: The race condition issue was fixed in FLUME-1236. If you pull in the latest sources, you would not see the race condition between channel start and create transaction. The issue Juhani has documented here is different from that. I can see in his logs, the log messages added by the fix for FLUME-1236.
Will: Could you please post the full logs, on FLUME-1232 and add a comment there? It will be easier to track that way.
was (Author: hshreedharan):
Inder: The race condition issue was fixed in FLUME-1236. If you pull in the latest sources, you would not see the race condition between channel start and create transaction.
Will: Could you please post the full logs, on FLUME-1232 and add a comment there? It will be easier to track that way.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286310#comment-13286310 ]
Hari Shreedharan commented on FLUME-1246:
-----------------------------------------
Juhani,
File channel failing to start - It looks like it is caused by the fix for FLUME-1236. The channel is not really failing to start, it is replaying events from the log. Now the agent fully starts up only once the channel is done starting(and replaying events from the log). So if there are a large number of events being replayed, you are likely to see a replay handler message in the flume log. Once it is done, the channel will start up. Can you please try waiting for a while and see if the agent starts up completely?
I have noticed the channel not stopping, probably during the replay phase. We should probably investigate this. I guess the channel eats interrupts during this phase.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287123#comment-13287123 ]
Hari Shreedharan commented on FLUME-1246:
-----------------------------------------
git log
commit e9d4fce54b61d8c0043452b422cf0d0c7840771e
Author: Hari Shreedharan <hs...@apache.org>
Date: Thu May 31 22:28:39 2012 +0000
FLUME-1215. Implement Timestamp interceptor
(Mike Percy via Hari Shreedharan)
git-svn-id: https://svn.apache.org/repos/asf/incubator/flume/trunk@1344932 13f79535-47bb-0310-9956-ffa450edef68
I ran an agent with SEQ first and then with NETCAT source. As you can see the channel starts, writes the seq source's output and then replays the remaining events when a new agent is started and writes the NETCAT events.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log, flume.log, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1246) FileChannel has a dependence on
hadoop IO and will without logs, making clean shutdown impossible
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy updated FLUME-1246:
------------------------------
Priority: Blocker (was: Major)
Fix Version/s: v1.2.0
Tentatively marking as v1.2.0 blocker per discussion on flume-dev
> FileChannel has a dependence on hadoop IO and will without logs, making clean shutdown impossible
> -------------------------------------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Priority: Blocker
> Fix For: v1.2.0
>
> Attachments: FLUME-1246.patch, flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> FileChannels Event implementation uses Hadoop IO's Writable for serialization. Without it a class loader error is thrown but never caught and logged killing the thread invisibly. This repeats and puts flume into a state where interrupts are not responded to, making clean shutdown impossible
> old desc for posterity:
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287273#comment-13287273 ]
Juhani Connolly commented on FLUME-1246:
----------------------------------------
Finally found the cause after adding a generic try/catch throwable block around start and logging it:
2012-06-01 17:38:29,787 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.channel.file.FileChannel.start(FileChannel.java:177)] Updated Starting FileChannel with dataDir [/home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata]
2012-06-01 17:38:29,790 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.channel.file.FileChannel.start(FileChannel.java:220)] failed
java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:179)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:228)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 23 more
with the extra code for the catch, FileChannel.java:179 corresponds to:
log = new Log(checkpointInterval, maxFileSize, capacity,
checkpointDir, dataDirs);"
Building on a system with hadoop installed and then deploying on a system without it. Since this is an Error, it passes invisibly through the lifecycle supervisor and the way things are now, fails without warning. This repeats constantly, and the supervisor doesn't receive interrupted exceptions.
I'm not sure why it's trying to load Writable at this point.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287075#comment-13287075 ]
Juhani Connolly commented on FLUME-1246:
----------------------------------------
yeah, the timetamp interceptor from mine is actually also a custom one, sorry. I did think afterwards it may be somehow related so removed it, but the problem persisted.
I'm going to try to track down where the exceptions aren't getting logged and work from there
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (FLUME-1246) FileChannel failing to start,
also shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287151#comment-13287151 ]
Hari Shreedharan edited comment on FLUME-1246 at 6/1/12 5:16 AM:
-----------------------------------------------------------------
I took the configuration(slightly modified though) you provided above and ran an agent:
Juhani,
I just took your configuration and made slight modifications:
test.channels.ch1.type = file
test.sources.top.type = exec
test.channels.ch1.checkpointDir = /Users/hshreedharan/work/flume-latest/flume/flume-ng-dist/target/flume-1.2.0-incubating-SNAPSHOT/bin/check
test.channels.ch1.dataDirs = /Users/hshreedharan/work/flume-latest/flume/flume-ng-dist/target/flume-1.2.0-incubating-SNAPSHOT/bin/filechdata
test.sources.top.command = top -b -d 1
test.sources.top.restart = true
test.sources.top.restartThrottle = 1000
test.sources.top.channels = ch1
test.sinks.log.type = logger
test.sinks.log.channel = ch1
test.channels = ch1
test.sources = top
test.sinks = log
I don't see an issue in the channel. I do see something with the exec source not being able to handle top, but the channel starts fine.
Edit: Please see latest flume.log
was (Author: hshreedharan):
I took the configuration(slightly modified though) you provided above and ran an agent:
Juhani,
I just took your configuration and made slight modifications:
test.channels.ch1.type = file
test.sources.top.type = exec
test.channels.ch1.checkpointDir = /Users/hshreedharan/work/flume-latest/flume/flume-ng-dist/target/flume-1.2.0-incubating-SNAPSHOT/bin/check
test.channels.ch1.dataDirs = /Users/hshreedharan/work/flume-latest/flume/flume-ng-dist/target/flume-1.2.0-incubating-SNAPSHOT/bin/filechdata
test.sources.top.command = top -b -d 1
test.sources.top.restart = true
test.sources.top.restartThrottle = 1000
test.sources.top.channels = ch1
test.sinks.log.type = logger
test.sinks.log.channel = ch1
test.channels = ch1
test.sources = top
test.sinks = log
I don't see an issue in the channel. I do see something with the exec source not being able to handle top, but the channel starts fine.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287481#comment-13287481 ]
Hari Shreedharan commented on FLUME-1246:
-----------------------------------------
Juhani:
The same issue can be found in RecoverableMemoryChannel and HDFSEventSink. I was about to suggest this, but thought you had Hadoop in the class path. The reason it tries to load is because it tries to create the class object for FileChannel(using Class.forName()), during which it tries to load all its internal member classes.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Inder SIngh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286386#comment-13286386 ]
Inder SIngh commented on FLUME-1246:
------------------------------------
I am hitting the same issue - Channel not starting.(easily reproducible)
Configuration
--------------
# Define a file channel called ch1 on agent1
agent1.channels.ch1.type = file
# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.bind = 0.0.0.0
agent1.sources.avro-source1.port = 41414
# Define a logger sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.log-sink1.channel = ch1
agent1.sinks.log-sink1.type = logger
# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = log-sink1
Enabled DEBUG logs and subset of interest is ->
2012-05-31 12:39:42,513 INFO file.FileChannel: Starting FileChannel with dataDir [/Users/inderbir.singh/.flume/file-channel/data]
2012-05-31 12:39:42,513 INFO nodemanager.DefaultLogicalNodeManager: Starting Source avro-source1
2012-05-31 12:39:42,514 INFO source.AvroSource: Avro source starting:AvroSource: { bindAddress:0.0.0.0 port:41414 }
2012-05-31 12:39:42,516 DEBUG flume.SinkRunner: Polling sink runner starting
2012-05-31 12:39:42,521 ERROR flume.SinkRunner: Unhandled exception, logging and sleeping for 5000ms
java.lang.IllegalStateException: Channel closed
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:237)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:118)
at org.apache.flume.sink.LoggerSink.process(LoggerSink.java:61)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:680)
2012-05-31 12:39:42,885 DEBUG source.AvroSource: Avro source started
Triage
-------
FileChannel.start() makes open = true; it's being called as the log shows - "Starting FileChannel with dataDir "
Looks like before the channel is started i.e. start() is finished LoggerSink is calling createTransaction -> where Preconditions.checkState(open, "Channel closed"); finds open=false since start() wasn't complete and fails.
If my understanding is correct this is a race condition.
Sources/Sinks should allow the channel's to be started before proceeding with them.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Juhani Connolly updated FLUME-1246:
-----------------------------------
Attachment: flume.log.20120601
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-1246:
------------------------------------
Attachment: flume.log
I took the configuration(slightly modified though) you provided above and ran an agent:
Juhani,
I just took your configuration and made slight modifications:
test.channels.ch1.type = file
test.sources.top.type = exec
test.channels.ch1.checkpointDir = /Users/hshreedharan/work/flume-latest/flume/flume-ng-dist/target/flume-1.2.0-incubating-SNAPSHOT/bin/check
test.channels.ch1.dataDirs = /Users/hshreedharan/work/flume-latest/flume/flume-ng-dist/target/flume-1.2.0-incubating-SNAPSHOT/bin/filechdata
test.sources.top.command = top -b -d 1
test.sources.top.restart = true
test.sources.top.restartThrottle = 1000
test.sources.top.channels = ch1
test.sinks.log.type = logger
test.sinks.log.channel = ch1
test.channels = ch1
test.sources = top
test.sinks = log
I don't see an issue in the channel. I do see something with the exec source not being able to handle top, but the channel starts fine.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Juhani Connolly updated FLUME-1246:
-----------------------------------
Attachment: flume.log
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286857#comment-13286857 ]
Hari Shreedharan commented on FLUME-1246:
-----------------------------------------
Inder: The race condition issue was fixed in FLUME-1236. If you pull in the latest sources, you would not see the race condition between channel start and create transaction.
Will: Could you please post the full logs, on FLUME-1232 and add a comment there? It will be easier to track that way.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287155#comment-13287155 ]
Hari Shreedharan commented on FLUME-1246:
-----------------------------------------
In the latest log you posted, the agent seems to have been killed before all components were even created, so start was never called:
2012-06-01 11:30:17,098 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [test]
2012-06-01 11:30:17,098 INFO properties.PropertiesFileConfigurationProvider: Creating channels
2012-06-01 11:30:17,098 DEBUG channel.DefaultChannelFactory: Creating instance of channel ch1 type file
2012-06-01 11:30:21,332 INFO node.FlumeNode: Flume node stopping - test
2012-06-01 11:30:21,332 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 9
2012-06-01 11:30:21,333 INFO nodemanager.DefaultLogicalNodeManager: Node manager stopping
2012-06-01 11:30:21,333 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 9
2012-06-01 11:30:21,334 DEBUG nodemanager.DefaultLogicalNodeManager: Node manager stopped
11:30:17 - channel factory log message
11:30:21 - shutdown.
I don't think all the components were created here, so the starts were never called.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287192#comment-13287192 ]
Juhani Connolly commented on FLUME-1246:
----------------------------------------
top worked fine with ExecSource for me, I had it running through a MemoryChannel no problem.
Yesterdays issue also could only be reproduced on some systems so I'm not too surprised that this config works in some places.
However even with full debug output it's not providing any clues as to what is wrong(with either yesterday or todays builds/problems)
You can notice from my log there is 4 seconds between channel creation and the stop. It may not seem like long but it should be enough. I've repeated this with much longer pauses before stopping it, and can post more logs if needed.
You can note from your logs here:
2012-05-31 22:12:07,148 INFO properties.PropertiesFileConfigurationProvider: Creating channels
2012-05-31 22:12:07,153 INFO properties.PropertiesFileConfigurationProvider: created channel ch1
that the channel creation is taking virtually no time.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-1246:
------------------------------------
Attachment: test_conf.conf
Conf I used, alternating between SEQ and NETCAT
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log, flume.log, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (FLUME-1246) FileChannel failing to start,
also shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287192#comment-13287192 ]
Juhani Connolly edited comment on FLUME-1246 at 6/1/12 5:46 AM:
----------------------------------------------------------------
top worked fine with ExecSource for me, I had it running through a MemoryChannel no problem.
Yesterdays issue also could only be reproduced on some systems so I'm not too surprised that this config works in some places.
However even with full debug output it's not providing any clues as to what is wrong(with either yesterday or todays builds/problems)
You can notice from my log there is 4 seconds between channel creation and the stop. It may not seem like long but it should be enough. I've repeated this with much longer pauses before stopping it, and can post more logs if needed.
edit: oops took the wrong part from your logs... The part with creating instance of filechannel appears to not appear in them, presumably because of the difference in cutoff point
was (Author: juhanic):
top worked fine with ExecSource for me, I had it running through a MemoryChannel no problem.
Yesterdays issue also could only be reproduced on some systems so I'm not too surprised that this config works in some places.
However even with full debug output it's not providing any clues as to what is wrong(with either yesterday or todays builds/problems)
You can notice from my log there is 4 seconds between channel creation and the stop. It may not seem like long but it should be enough. I've repeated this with much longer pauses before stopping it, and can post more logs if needed.
You can note from your logs here:
2012-05-31 22:12:07,148 INFO properties.PropertiesFileConfigurationProvider: Creating channels
2012-05-31 22:12:07,153 INFO properties.PropertiesFileConfigurationProvider: created channel ch1
that the channel creation is taking virtually no time.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel hangs silently when
Hadoop libs not found
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396456#comment-13396456 ]
Hudson commented on FLUME-1246:
-------------------------------
Integrated in flume-trunk #239 (See [https://builds.apache.org/job/flume-trunk/239/])
FLUME-1246: FileChannel hangs silently when Hadoop libs not found
(Hari Shreedharan via Juhani Connolly) (Revision 1351553)
Result = SUCCESS
juhanic : http://svn.apache.org/viewvc/?view=rev&rev=1351553
Files :
* /incubator/flume/trunk/flume-ng-node/src/main/java/org/apache/flume/conf/file/AbstractFileConfigurationProvider.java
> FileChannel hangs silently when Hadoop libs not found
> -----------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Priority: Blocker
> Fix For: v1.2.0
>
> Attachments: FLUME-1246.patch, flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> FileChannels Event implementation uses Hadoop IO's Writable for serialization. Without it a class loader error is thrown but never caught and logged killing the thread invisibly. This repeats and puts flume into a state where interrupts are not responded to, making clean shutdown impossible
> old desc for posterity:
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (FLUME-1246) FileChannel failing to start,
also shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287205#comment-13287205 ]
Juhani Connolly edited comment on FLUME-1246 at 6/1/12 6:49 AM:
----------------------------------------------------------------
I've rebuilt everything with mvn clean package and redeployed just in case and things still aren't changing. I can run it and not shutdown, and it will just stay inactive after it hits Creating instance of filechannel. I'm going to try and add some extra traces and take a stack dump, and I'll get back to this hopefully with some more useful info as it doesn't look like you can recreate it. I'm guessing it's some issue with our setup and that the necessary error messages aren't being generated.
Premature incorrect assumption on that last edit sorry, going to take the time to just trawl this thoroughly and then get back once I've fixed it
was (Author: juhanic):
I've rebuilt everything with mvn clean package and redeployed just in case and things still aren't changing. I can run it and not shutdown, and it will just stay inactive after it hits Creating instance of filechannel. I'm going to try and add some extra traces and take a stack dump, and I'll get back to this hopefully with some more useful info as it doesn't look like you can recreate it. I'm guessing it's some issue with our setup and that the necessary error messages aren't being generated.
edit: Turns out that my new problem is with mvn clean package not actually putting the file channel into the tar binary for distribution(even though mvn is giving a success: [INFO] Flume NG file-based channel ....................... SUCCESS [2.901s]), and also with the FlumeException that should be reporting the missing class getting swallowed up, 2 separate new issues I guess.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286312#comment-13286312 ]
Juhani Connolly commented on FLUME-1246:
----------------------------------------
It's an empty channel on a fresh server, there is no data to replay. It works fine on some machine not others, so I assume it's some setup specific weirdness. The issue really is there are no relevant logs to debug it, and the shutdown altogether not working.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-1246:
------------------------------------
Attachment: (was: flume.log)
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (FLUME-1246) FileChannel failing to start,
also shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287205#comment-13287205 ]
Juhani Connolly edited comment on FLUME-1246 at 6/1/12 6:40 AM:
----------------------------------------------------------------
I've rebuilt everything with mvn clean package and redeployed just in case and things still aren't changing. I can run it and not shutdown, and it will just stay inactive after it hits Creating instance of filechannel. I'm going to try and add some extra traces and take a stack dump, and I'll get back to this hopefully with some more useful info as it doesn't look like you can recreate it. I'm guessing it's some issue with our setup and that the necessary error messages aren't being generated.
edit: Turns out that my new problem is with mvn clean package not actually putting the file channel into the tar binary for distribution(even though mvn is giving a success: [INFO] Flume NG file-based channel ....................... SUCCESS [2.901s]), and also with the FlumeException that should be reporting the missing class getting swallowed up, 2 separate new issues I guess.
was (Author: juhanic):
I've rebuilt everything with mvn clean package and redeployed just in case and things still aren't changing. I can run it and not shutdown, and it will just stay inactive after it hits Creating instance of filechannel. I'm going to try and add some extra traces and take a stack dump, and I'll get back to this hopefully with some more useful info as it doesn't look like you can recreate it. I'm guessing it's some issue with our setup and that the necessary error messages aren't being generated.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287205#comment-13287205 ]
Juhani Connolly commented on FLUME-1246:
----------------------------------------
I've rebuilt everything with mvn clean package and redeployed just in case and things still aren't changing. I can run it and not shutdown, and it will just stay inactive after it hits Creating instance of filechannel. I'm going to try and add some extra traces and take a stack dump, and I'll get back to this hopefully with some more useful info as it doesn't look like you can recreate it. I'm guessing it's some issue with our setup and that the necessary error messages aren't being generated.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (FLUME-1246) FileChannel failing to start,
also shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286310#comment-13286310 ]
Hari Shreedharan edited comment on FLUME-1246 at 5/31/12 4:40 AM:
------------------------------------------------------------------
Juhani,
File channel failing to start - It looks like it is caused by the fix for FLUME-1236. The channel is not really failing to start, it is replaying events from the log. Now the agent fully starts up only once the channel is done starting(and replaying events from the log). So if there are a large number of events being replayed, you are likely to see a replay handler message in the flume log. Once it is done, the channel will start up. Can you please try waiting for a while and see if the agent starts up completely? The only reason I am not fully convinced it is FLUME-1236 fix, is that I do not see any replay handler messages in the log you posted. So it would be great if you could wait for a while and see if the agent starts up completely.
I have noticed the channel not stopping(and had to resort to kill -9), probably during the replay phase. We should probably investigate this. I guess the channel eats interrupts during this phase.
was (Author: hshreedharan):
Juhani,
File channel failing to start - It looks like it is caused by the fix for FLUME-1236. The channel is not really failing to start, it is replaying events from the log. Now the agent fully starts up only once the channel is done starting(and replaying events from the log). So if there are a large number of events being replayed, you are likely to see a replay handler message in the flume log. Once it is done, the channel will start up. Can you please try waiting for a while and see if the agent starts up completely?
I have noticed the channel not stopping, probably during the replay phase. We should probably investigate this. I guess the channel eats interrupts during this phase.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1246) FileChannel has a dependence on
hadoop IO and will without logs, making clean shutdown impossible
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-1246:
------------------------------------
Attachment: FLUME-1246.patch
Stop gap fix till FileChannel dependency issues are resolved, to not cause the agent to go into an undefined state.
> FileChannel has a dependence on hadoop IO and will without logs, making clean shutdown impossible
> -------------------------------------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: FLUME-1246.patch, flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> FileChannels Event implementation uses Hadoop IO's Writable for serialization. Without it a class loader error is thrown but never caught and logged killing the thread invisibly. This repeats and puts flume into a state where interrupts are not responded to, making clean shutdown impossible
> old desc for posterity:
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1246) FileChannel failing to start, also
shutdown impossible without kill
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287222#comment-13287222 ]
Juhani Connolly commented on FLUME-1246:
----------------------------------------
Ok, the new problem was with packaging, instantiating FileChannel was not working due to compilation errors which were throwing an Error(which is not caught anywhere and thus causes everything to fail transparently). I was under the impression that running "mvn clean package" should be enough(please correct me if this assumption is incorrect. If it's correct, I suspect I have to fiddle with eclipse), but after running clean, compile, install, package individually in order, the new problem is fixed and I'm back to the original problem. My apologies for the distraction.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume-hari-2012May312217PST.log, flume.log, flume.log, flume.log, flume.log.20120601, test_conf.conf
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (FLUME-1246) FileChannel failing to start,
also shutdown impossible without kill
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287118#comment-13287118 ]
Hari Shreedharan edited comment on FLUME-1246 at 6/1/12 3:09 AM:
-----------------------------------------------------------------
Juhani,
I am fairly sure start is being called. From the log you posted earlier today:
2012-05-31 12:39:42,512 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel ch1
and the earlier log too:
2012-05-31 12:51:15,100 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel ch1
This is from here(DefaultLogicalNodeManager#startAllComponents):
for (Entry<String, Channel> entry :
nodeConfiguration.getChannels().entrySet()) {
try{
logger.info("Starting Channel " + entry.getKey());
nodeSupervisor.supervise(entry.getValue(),
new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
} catch (Exception e){
logger.error("Error while starting {}", entry.getValue(), e);
}
}
The channel failing to start does not seem related to start not getting called. If you try running it through a debugger and put a break point at start you can verify.
was (Author: hshreedharan):
Juhani,
I am fairly sure start is being called. From the log you posted earlier today:
2012-05-31 12:39:42,512 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel ch1
and the earlier log too:
2012-05-31 12:51:15,100 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel ch1
This is from here:
for (Entry<String, Channel> entry :
nodeConfiguration.getChannels().entrySet()) {
try{
logger.info("Starting Channel " + entry.getKey());
nodeSupervisor.supervise(entry.getValue(),
new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
} catch (Exception e){
logger.error("Error while starting {}", entry.getValue(), e);
}
}
The channel failing to start does not seem related to start not getting called. If you try running it through a debugger and put a break point at start you can verify.
> FileChannel failing to start, also shutdown impossible without kill
> --------------------------------------------------------------------
>
> Key: FLUME-1246
> URL: https://issues.apache.org/jira/browse/FLUME-1246
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Environment: CentOS 5.4
> Reporter: Juhani Connolly
> Attachments: flume.log, flume.log
>
>
> Reduced to a minimal configuration for simplicity. I can recreate this on some machines, and not others. I wouldn't be surprised if it is some machines specific issue(test machines on CentOS5.4. On some it worked, others not), however whatever exception was thrown when attempting to get created is consumed and never passed onwards
> Config:
> test.channels.ch1.type = file
> test.channels.ch1.checkpointDir = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/check
> test.channels.ch1.dataDirs = /home/share/juhani_connolly/flume-1.2.0-incubating-SNAPSHOT/filechdata
> test.sources.top.type = exec
> test.sources.top.command = /usr/bin top -b -d 1
> test.sources.top.restart = true
> test.sources.top.restartThrottle = 1000
> test.sources.top.interceptors = ts
> test.sources.top.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
> test.sources.top.channels = ch1
> test.sinks.log.type = logger
> test.sinks.log.channel = ch1
> test.channels = ch1
> test.sources = top
> test.sinks = log
> attaching logs with general/lifecycle loglevel down to debug.
> A solution to this is probably going to be just improving error reporting.
> Another possibly more important element is that flume enters a state from which it cannot shutdown without kill -9. It looks like the Interrupts are getting swallowed up silently somewhere
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira