You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Josh Marcus <jm...@meetup.com> on 2014/01/10 00:50:58 UTC
issues with configuration updates and clean shutdowns
Hey folks,
I'm setting up new flume agents, and I'm noticing issues with both live
updates to the configuration and cleanly shutting down an agent. I was
wondering if anyone should shed light on whether what I'm seeing is based
on known issues or if there are any likely culprits.
First of all, if I update the configuration, the agent notices the updated
configuation, shuts down (successfully) but doesn't restart with the
updated configuration.
Secondly, sending a signal (I've tried a variety) to a running flume agent
doesn't cleanly shutdown the agent -- for example, the .tmp output file
doesn't get moved to its permanent location.
So, if I update the configuration (or just touch it), I get a clean
shutdown -- but it won't restart. But I don't currently have alternate way
that will reliably shut it down.
Any advice?
--j
Re: issues with configuration updates and clean shutdowns
Posted by Brock Noland <br...@cloudera.com>.
Could you share some more information? For example:
* What version you are using
* Configuration file
* Log
On Wed, Jan 15, 2014 at 8:24 AM, Josh Marcus <jm...@meetup.com> wrote:
> Hey folks,
>
> I didn't get any response at all to this, so let me ask more generally:
> does automatic reload upon config change generally work? Does
> sending a signal for shutdown or reload generally work? Is this just a
> symptom of our configuration?
>
> --j
>
>
>
> On Thu, Jan 9, 2014 at 6:50 PM, Josh Marcus <jm...@meetup.com> wrote:
>
>> Hey folks,
>>
>> I'm setting up new flume agents, and I'm noticing issues with both live
>> updates to the configuration and cleanly shutting down an agent. I was
>> wondering if anyone should shed light on whether what I'm seeing is based
>> on known issues or if there are any likely culprits.
>>
>> First of all, if I update the configuration, the agent notices the
>> updated configuation, shuts down (successfully) but doesn't restart with
>> the updated configuration.
>>
>> Secondly, sending a signal (I've tried a variety) to a running flume
>> agent doesn't cleanly shutdown the agent -- for example, the .tmp output
>> file doesn't get moved to its permanent location.
>>
>> So, if I update the configuration (or just touch it), I get a clean
>> shutdown -- but it won't restart. But I don't currently have alternate way
>> that will reliably shut it down.
>>
>> Any advice?
>>
>> --j
>>
>>
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: issues with configuration updates and clean shutdowns
Posted by Josh Marcus <jm...@meetup.com>.
Hey folks,
Any thoughts now that I've posted a config and logs?
--j
On Wed, Jan 15, 2014 at 10:23 AM, Josh Marcus <jm...@meetup.com> wrote:
> Brock & Jeff:
>
> Thanks so much for your questions. I've created a gist of an agent with a
> simple config that demonstrates the issue and accompanying log files.
>
> https://gist.github.com/joshmarcus/8438107
>
> I'm using flume 1.4.0.
>
> The logs describe a scenario where flume is started, I send log messages,
> and then I touch the config (no modifications except for modification
> time). If I edit the config, the situation is the same, but the touch is
> to try and minimize this test scenario. The HDFS file is renamed from
> .tmp, but future messages are not logged.
>
> I experimented with sending TERM (for shutdown) and HUP (for potential
> reload) but neither work, but I haven't included logs for those situations
> (as that's not what I'm actually trying to do here).
>
> --j
>
>
> On Wed, Jan 15, 2014 at 9:39 AM, Jeff Lord <jl...@cloudera.com> wrote:
>
>> Josh,
>>
>> If you modify your config than the flume agent will see the config has
>> changed and reload any components that have been modified. Are you able to
>> provide the logs from the flume agent which occur following a modification
>> of the config? What command are you using to signal for a shutdown?
>>
>> -Jeff
>>
>>
>> On Wed, Jan 15, 2014 at 6:24 AM, Josh Marcus <jm...@meetup.com> wrote:
>>
>>> Hey folks,
>>>
>>> I didn't get any response at all to this, so let me ask more generally:
>>> does automatic reload upon config change generally work? Does
>>> sending a signal for shutdown or reload generally work? Is this just a
>>> symptom of our configuration?
>>>
>>> --j
>>>
>>>
>>>
>>> On Thu, Jan 9, 2014 at 6:50 PM, Josh Marcus <jm...@meetup.com> wrote:
>>>
>>>> Hey folks,
>>>>
>>>> I'm setting up new flume agents, and I'm noticing issues with both live
>>>> updates to the configuration and cleanly shutting down an agent. I was
>>>> wondering if anyone should shed light on whether what I'm seeing is based
>>>> on known issues or if there are any likely culprits.
>>>>
>>>> First of all, if I update the configuration, the agent notices the
>>>> updated configuation, shuts down (successfully) but doesn't restart with
>>>> the updated configuration.
>>>>
>>>> Secondly, sending a signal (I've tried a variety) to a running flume
>>>> agent doesn't cleanly shutdown the agent -- for example, the .tmp output
>>>> file doesn't get moved to its permanent location.
>>>>
>>>> So, if I update the configuration (or just touch it), I get a clean
>>>> shutdown -- but it won't restart. But I don't currently have alternate way
>>>> that will reliably shut it down.
>>>>
>>>> Any advice?
>>>>
>>>> --j
>>>>
>>>>
>>>
>>
>
Re: issues with configuration updates and clean shutdowns
Posted by Josh Marcus <jm...@meetup.com>.
Brock & Jeff:
Thanks so much for your questions. I've created a gist of an agent with a
simple config that demonstrates the issue and accompanying log files.
https://gist.github.com/joshmarcus/8438107
I'm using flume 1.4.0.
The logs describe a scenario where flume is started, I send log messages,
and then I touch the config (no modifications except for modification
time). If I edit the config, the situation is the same, but the touch is
to try and minimize this test scenario. The HDFS file is renamed from
.tmp, but future messages are not logged.
I experimented with sending TERM (for shutdown) and HUP (for potential
reload) but neither work, but I haven't included logs for those situations
(as that's not what I'm actually trying to do here).
--j
On Wed, Jan 15, 2014 at 9:39 AM, Jeff Lord <jl...@cloudera.com> wrote:
> Josh,
>
> If you modify your config than the flume agent will see the config has
> changed and reload any components that have been modified. Are you able to
> provide the logs from the flume agent which occur following a modification
> of the config? What command are you using to signal for a shutdown?
>
> -Jeff
>
>
> On Wed, Jan 15, 2014 at 6:24 AM, Josh Marcus <jm...@meetup.com> wrote:
>
>> Hey folks,
>>
>> I didn't get any response at all to this, so let me ask more generally:
>> does automatic reload upon config change generally work? Does
>> sending a signal for shutdown or reload generally work? Is this just a
>> symptom of our configuration?
>>
>> --j
>>
>>
>>
>> On Thu, Jan 9, 2014 at 6:50 PM, Josh Marcus <jm...@meetup.com> wrote:
>>
>>> Hey folks,
>>>
>>> I'm setting up new flume agents, and I'm noticing issues with both live
>>> updates to the configuration and cleanly shutting down an agent. I was
>>> wondering if anyone should shed light on whether what I'm seeing is based
>>> on known issues or if there are any likely culprits.
>>>
>>> First of all, if I update the configuration, the agent notices the
>>> updated configuation, shuts down (successfully) but doesn't restart with
>>> the updated configuration.
>>>
>>> Secondly, sending a signal (I've tried a variety) to a running flume
>>> agent doesn't cleanly shutdown the agent -- for example, the .tmp output
>>> file doesn't get moved to its permanent location.
>>>
>>> So, if I update the configuration (or just touch it), I get a clean
>>> shutdown -- but it won't restart. But I don't currently have alternate way
>>> that will reliably shut it down.
>>>
>>> Any advice?
>>>
>>> --j
>>>
>>>
>>
>
Re: issues with configuration updates and clean shutdowns
Posted by Jeff Lord <jl...@cloudera.com>.
Josh,
If you modify your config than the flume agent will see the config has
changed and reload any components that have been modified. Are you able to
provide the logs from the flume agent which occur following a modification
of the config? What command are you using to signal for a shutdown?
-Jeff
On Wed, Jan 15, 2014 at 6:24 AM, Josh Marcus <jm...@meetup.com> wrote:
> Hey folks,
>
> I didn't get any response at all to this, so let me ask more generally:
> does automatic reload upon config change generally work? Does
> sending a signal for shutdown or reload generally work? Is this just a
> symptom of our configuration?
>
> --j
>
>
>
> On Thu, Jan 9, 2014 at 6:50 PM, Josh Marcus <jm...@meetup.com> wrote:
>
>> Hey folks,
>>
>> I'm setting up new flume agents, and I'm noticing issues with both live
>> updates to the configuration and cleanly shutting down an agent. I was
>> wondering if anyone should shed light on whether what I'm seeing is based
>> on known issues or if there are any likely culprits.
>>
>> First of all, if I update the configuration, the agent notices the
>> updated configuation, shuts down (successfully) but doesn't restart with
>> the updated configuration.
>>
>> Secondly, sending a signal (I've tried a variety) to a running flume
>> agent doesn't cleanly shutdown the agent -- for example, the .tmp output
>> file doesn't get moved to its permanent location.
>>
>> So, if I update the configuration (or just touch it), I get a clean
>> shutdown -- but it won't restart. But I don't currently have alternate way
>> that will reliably shut it down.
>>
>> Any advice?
>>
>> --j
>>
>>
>
Re: issues with configuration updates and clean shutdowns
Posted by Josh Marcus <jm...@meetup.com>.
Hey folks,
I didn't get any response at all to this, so let me ask more generally:
does automatic reload upon config change generally work? Does
sending a signal for shutdown or reload generally work? Is this just a
symptom of our configuration?
--j
On Thu, Jan 9, 2014 at 6:50 PM, Josh Marcus <jm...@meetup.com> wrote:
> Hey folks,
>
> I'm setting up new flume agents, and I'm noticing issues with both live
> updates to the configuration and cleanly shutting down an agent. I was
> wondering if anyone should shed light on whether what I'm seeing is based
> on known issues or if there are any likely culprits.
>
> First of all, if I update the configuration, the agent notices the updated
> configuation, shuts down (successfully) but doesn't restart with the
> updated configuration.
>
> Secondly, sending a signal (I've tried a variety) to a running flume agent
> doesn't cleanly shutdown the agent -- for example, the .tmp output file
> doesn't get moved to its permanent location.
>
> So, if I update the configuration (or just touch it), I get a clean
> shutdown -- but it won't restart. But I don't currently have alternate way
> that will reliably shut it down.
>
> Any advice?
>
> --j
>
>