You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flume.apache.org by David Capwell <dc...@gmail.com> on 2012/08/13 20:33:15 UTC

dynamic hostnames for sinks

I am looking at flume for data ingestion on AWS.  I was wondering how
people handle dynamic hostnames in flume?

I have created my own ConfigurationProvider and looking to
mimic FailoverSinkProcessor but on failover to dynamically find a new host
to talk to.

Is there a better route to go than the one I went down?

Thanks for your time reading this email.

Re: dynamic hostnames for sinks

Posted by David Capwell <dc...@gmail.com>.

What I did was modified the agent to register itself with ZK if its source
listens to a port.  I then have it setup so that the the agent will check
ZK if the sink defines a config called "cluster".  This is used to discover
the host to talk to.  If the sink defines cluster, the host/port will be
replaced.

This was all done in the agent itself so i could inherit all source/sinks
out there.

I was thinking of forking the FailoverSinkProcessor so on failover discover
a new host to talk to and create a new sink.

On Mon, Aug 13, 2012 at 1:11 PM, Ralph Goers <ra...@dslextreme.com>wrote:

> I'd be interested in knowing how you dynamically located the new host.  I
> would also imagine you could create a "smart" sink that could failover
> itself.
>
> Ralph
>
> On Aug 13, 2012, at 11:33 AM, David Capwell wrote:
>
> > I am looking at flume for data ingestion on AWS.  I was wondering how
> people handle dynamic hostnames in flume?
> >
> > I have created my own ConfigurationProvider and looking to mimic
> FailoverSinkProcessor but on failover to dynamically find a new host to
> talk to.
> >
> > Is there a better route to go than the one I went down?
> >
> > Thanks for your time reading this email.
>
>

Re: dynamic hostnames for sinks

Posted by Ralph Goers <ra...@dslextreme.com>.

I'd be interested in knowing how you dynamically located the new host.  I would also imagine you could create a "smart" sink that could failover itself.

Ralph

On Aug 13, 2012, at 11:33 AM, David Capwell wrote:

> I am looking at flume for data ingestion on AWS.  I was wondering how people handle dynamic hostnames in flume?
> 
> I have created my own ConfigurationProvider and looking to mimic FailoverSinkProcessor but on failover to dynamically find a new host to talk to.
> 
> Is there a better route to go than the one I went down?
> 
> Thanks for your time reading this email.