You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Inder Pall <in...@gmail.com> on 2012/07/12 16:04:31 UTC
Configuring FLUME to use FailOverSinkProcessor...
Folks,
for some reason updating the JIRA isn't trigerring an email....
I need feedback from FLUME DEVS on
FLUME-1045<https://issues.apache.org/jira/browse/FLUME-1045?focusedCommentId=13412737&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13412737>
wherein
i am trying to use failover sinkprocessor and a combination of file and
memory channel to achieve scribe like spooling/de-spooling.....facing some
issues here....
--
Thanks,
- Inder
Tech Platforms @Inmobi
Linkedin - http://goo.gl/eR4Ub
Re: Configuring FLUME to use FailOverSinkProcessor...
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Hi Inder,
actually in my testing environment, events weren't drain from memory channel and therefore they were not saved in file channel. I'm not sure why, but it appeared that the failover did not happened as expected. I unfortunately did not yet have enough time to fully explore what has happen (I might be doing something wrong).
Jarcec
On Jul 13, 2012, at 3:51 PM, Inder Pall wrote:
> So if this thing works what are people's thoughts about using it for PROD
> envs...
>
> Jarek, the reason i had hdfs data sink locations same for both HDFS sinks
> was to ensure that the spooled data also makes to the final location...so
> the test to try out would be to check all numbers generated by the seqeunce
> generator source are available once the tests are finished...
>
> btw, the avro-sink & avro source combination for file channel is a little
> heavy as it's local..perhaps something lighter could be better.
>
> * I turned off name node, event's were not stored in file channel as
> expected
>>> here you meant they were indeed saved into file channel and drained from
> memory channel right??
>
> Thanks,
> - Inder
>
> On Fri, Jul 13, 2012 at 7:18 PM, Inder Pall <in...@gmail.com> wrote:
>
>> Jarek,
>>
>> thanks for taking out time to try this..yeah i meant mem channel to be
>> used first and then file channel for some reason i thought lower number
>> means higher priority pardon the ignorance of not looking at the
>> documentation.....
>>
>> i have a dated code-base of trunk around 10-12 days...the agent comes up
>> fine but the sequence generator source wasn't sending any events...or
>> actually i didn't see anything in the logs....
>>
>>
>
>
>> Thanks,
>> - Inder
>>
>>
>> On Fri, Jul 13, 2012 at 3:30 PM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
>>
>>> Hi Inder,
>>> could you please advise what exactly was (or wasn't) happening to you?
>>> I've tried your configuration file on current trunk and it was working for
>>> me out of the box.
>>>
>>> I was playing with it to see if your idea will work and I've ended up
>>> with attached configuration. It contains just few modification to yours:
>>>
>>> * I've swapped priorities to firstly use the memory channel and fail over
>>> to file channel
>>> * I've renamed target file prefixes to distinguish source of the events
>>> (memory or file channel)
>>>
>>> Scenario that I've run:
>>>
>>> * I turned on the flume agents, event's were correctly saved on HDFS
>>> * I turned off name node, event's were not stored in file channel as
>>> expected
>>>
>>> I'm not sure why the observed behaviour was different than expected and
>>> 'I'll investigate that later. Meantime, could you describe what exactly was
>>> happening to you?
>>>
>>> Jarcec
>>>
>>> On Thu, Jul 12, 2012 at 04:11:08PM +0200, Jarek Jarcec Cecho wrote:
>>>> I'm just looking on that sir.
>>>>
>>>> Jarcec
>>>>
>>>> On Thu, Jul 12, 2012 at 07:34:31PM +0530, Inder Pall wrote:
>>>>> Folks,
>>>>>
>>>>> for some reason updating the JIRA isn't trigerring an email....
>>>>> I need feedback from FLUME DEVS on
>>>>> FLUME-1045<
>>> https://issues.apache.org/jira/browse/FLUME-1045?focusedCommentId=13412737&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13412737
>>>>
>>>>> wherein
>>>>> i am trying to use failover sinkprocessor and a combination of file
>>> and
>>>>> memory channel to achieve scribe like spooling/de-spooling.....facing
>>> some
>>>>> issues here....
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> - Inder
>>>>> Tech Platforms @Inmobi
>>>>> Linkedin - http://goo.gl/eR4Ub
>>>
>>>
>>>
>>
>>
>> --
>> Thanks,
>> - Inder
>> Tech Platforms @Inmobi
>> Linkedin - http://goo.gl/eR4Ub
>>
>
>
>
> --
> Thanks,
> - Inder
> Tech Platforms @Inmobi
> Linkedin - http://goo.gl/eR4Ub
Re: Configuring FLUME to use FailOverSinkProcessor...
Posted by Inder Pall <in...@gmail.com>.
So if this thing works what are people's thoughts about using it for PROD
envs...
Jarek, the reason i had hdfs data sink locations same for both HDFS sinks
was to ensure that the spooled data also makes to the final location...so
the test to try out would be to check all numbers generated by the seqeunce
generator source are available once the tests are finished...
btw, the avro-sink & avro source combination for file channel is a little
heavy as it's local..perhaps something lighter could be better.
* I turned off name node, event's were not stored in file channel as
expected
>> here you meant they were indeed saved into file channel and drained from
memory channel right??
Thanks,
- Inder
On Fri, Jul 13, 2012 at 7:18 PM, Inder Pall <in...@gmail.com> wrote:
> Jarek,
>
> thanks for taking out time to try this..yeah i meant mem channel to be
> used first and then file channel for some reason i thought lower number
> means higher priority pardon the ignorance of not looking at the
> documentation.....
>
> i have a dated code-base of trunk around 10-12 days...the agent comes up
> fine but the sequence generator source wasn't sending any events...or
> actually i didn't see anything in the logs....
>
>
> Thanks,
> - Inder
>
>
> On Fri, Jul 13, 2012 at 3:30 PM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
>
>> Hi Inder,
>> could you please advise what exactly was (or wasn't) happening to you?
>> I've tried your configuration file on current trunk and it was working for
>> me out of the box.
>>
>> I was playing with it to see if your idea will work and I've ended up
>> with attached configuration. It contains just few modification to yours:
>>
>> * I've swapped priorities to firstly use the memory channel and fail over
>> to file channel
>> * I've renamed target file prefixes to distinguish source of the events
>> (memory or file channel)
>>
>> Scenario that I've run:
>>
>> * I turned on the flume agents, event's were correctly saved on HDFS
>> * I turned off name node, event's were not stored in file channel as
>> expected
>>
>> I'm not sure why the observed behaviour was different than expected and
>> 'I'll investigate that later. Meantime, could you describe what exactly was
>> happening to you?
>>
>> Jarcec
>>
>> On Thu, Jul 12, 2012 at 04:11:08PM +0200, Jarek Jarcec Cecho wrote:
>> > I'm just looking on that sir.
>> >
>> > Jarcec
>> >
>> > On Thu, Jul 12, 2012 at 07:34:31PM +0530, Inder Pall wrote:
>> > > Folks,
>> > >
>> > > for some reason updating the JIRA isn't trigerring an email....
>> > > I need feedback from FLUME DEVS on
>> > > FLUME-1045<
>> https://issues.apache.org/jira/browse/FLUME-1045?focusedCommentId=13412737&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13412737
>> >
>> > > wherein
>> > > i am trying to use failover sinkprocessor and a combination of file
>> and
>> > > memory channel to achieve scribe like spooling/de-spooling.....facing
>> some
>> > > issues here....
>> > >
>> > > --
>> > > Thanks,
>> > > - Inder
>> > > Tech Platforms @Inmobi
>> > > Linkedin - http://goo.gl/eR4Ub
>>
>>
>>
>
>
> --
> Thanks,
> - Inder
> Tech Platforms @Inmobi
> Linkedin - http://goo.gl/eR4Ub
>
--
Thanks,
- Inder
Tech Platforms @Inmobi
Linkedin - http://goo.gl/eR4Ub
Re: Configuring FLUME to use FailOverSinkProcessor...
Posted by Inder Pall <in...@gmail.com>.
Jarek,
thanks for taking out time to try this..yeah i meant mem channel to be used
first and then file channel for some reason i thought lower number means
higher priority pardon the ignorance of not looking at the
documentation.....
i have a dated code-base of trunk around 10-12 days...the agent comes up
fine but the sequence generator source wasn't sending any events...or
actually i didn't see anything in the logs....
Thanks,
- Inder
On Fri, Jul 13, 2012 at 3:30 PM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
> Hi Inder,
> could you please advise what exactly was (or wasn't) happening to you?
> I've tried your configuration file on current trunk and it was working for
> me out of the box.
>
> I was playing with it to see if your idea will work and I've ended up with
> attached configuration. It contains just few modification to yours:
>
> * I've swapped priorities to firstly use the memory channel and fail over
> to file channel
> * I've renamed target file prefixes to distinguish source of the events
> (memory or file channel)
>
> Scenario that I've run:
>
> * I turned on the flume agents, event's were correctly saved on HDFS
> * I turned off name node, event's were not stored in file channel as
> expected
>
> I'm not sure why the observed behaviour was different than expected and
> 'I'll investigate that later. Meantime, could you describe what exactly was
> happening to you?
>
> Jarcec
>
> On Thu, Jul 12, 2012 at 04:11:08PM +0200, Jarek Jarcec Cecho wrote:
> > I'm just looking on that sir.
> >
> > Jarcec
> >
> > On Thu, Jul 12, 2012 at 07:34:31PM +0530, Inder Pall wrote:
> > > Folks,
> > >
> > > for some reason updating the JIRA isn't trigerring an email....
> > > I need feedback from FLUME DEVS on
> > > FLUME-1045<
> https://issues.apache.org/jira/browse/FLUME-1045?focusedCommentId=13412737&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13412737
> >
> > > wherein
> > > i am trying to use failover sinkprocessor and a combination of file and
> > > memory channel to achieve scribe like spooling/de-spooling.....facing
> some
> > > issues here....
> > >
> > > --
> > > Thanks,
> > > - Inder
> > > Tech Platforms @Inmobi
> > > Linkedin - http://goo.gl/eR4Ub
>
>
>
--
Thanks,
- Inder
Tech Platforms @Inmobi
Linkedin - http://goo.gl/eR4Ub
Re: Configuring FLUME to use FailOverSinkProcessor...
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Hi Inder,
could you please advise what exactly was (or wasn't) happening to you? I've tried your configuration file on current trunk and it was working for me out of the box.
I was playing with it to see if your idea will work and I've ended up with attached configuration. It contains just few modification to yours:
* I've swapped priorities to firstly use the memory channel and fail over to file channel
* I've renamed target file prefixes to distinguish source of the events (memory or file channel)
Scenario that I've run:
* I turned on the flume agents, event's were correctly saved on HDFS
* I turned off name node, event's were not stored in file channel as expected
I'm not sure why the observed behaviour was different than expected and 'I'll investigate that later. Meantime, could you describe what exactly was happening to you?
Jarcec
On Thu, Jul 12, 2012 at 04:11:08PM +0200, Jarek Jarcec Cecho wrote:
> I'm just looking on that sir.
>
> Jarcec
>
> On Thu, Jul 12, 2012 at 07:34:31PM +0530, Inder Pall wrote:
> > Folks,
> >
> > for some reason updating the JIRA isn't trigerring an email....
> > I need feedback from FLUME DEVS on
> > FLUME-1045<https://issues.apache.org/jira/browse/FLUME-1045?focusedCommentId=13412737&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13412737>
> > wherein
> > i am trying to use failover sinkprocessor and a combination of file and
> > memory channel to achieve scribe like spooling/de-spooling.....facing some
> > issues here....
> >
> > --
> > Thanks,
> > - Inder
> > Tech Platforms @Inmobi
> > Linkedin - http://goo.gl/eR4Ub
Re: Configuring FLUME to use FailOverSinkProcessor...
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
I'm just looking on that sir.
Jarcec
On Thu, Jul 12, 2012 at 07:34:31PM +0530, Inder Pall wrote:
> Folks,
>
> for some reason updating the JIRA isn't trigerring an email....
> I need feedback from FLUME DEVS on
> FLUME-1045<https://issues.apache.org/jira/browse/FLUME-1045?focusedCommentId=13412737&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13412737>
> wherein
> i am trying to use failover sinkprocessor and a combination of file and
> memory channel to achieve scribe like spooling/de-spooling.....facing some
> issues here....
>
> --
> Thanks,
> - Inder
> Tech Platforms @Inmobi
> Linkedin - http://goo.gl/eR4Ub