Embedding Chukwa agent/adapter?

[Kirk True <ki...@mustardgrain.com>]

Hi all,

We'd like to push data to the collectors straight from our application 
as it receives data from its clients. In a sense we'd like to embed a 
Chukwa agent and our adapters inside our web application. Does this make 
sense or is this approach flawed in some way? Otherwise we'd have to 
save the data to a file in a shared directory that's then picked up by a 
stand-alone Chukwa agent via a 'file tailer' or something. Or do we want 
to use the UDPAdaptor and proxy our data to it, which in turn forwards 
it to the collector?

Thanks,
Kirk

Re: Embedding Chukwa agent/adapter?

[Ariel Rabkin <as...@gmail.com>]

Nothing unreasonable about that modified adaptor.  However, you
presumably want to delete the file, not after sending the chunk, but
after getting back the Ack.  You can implement the interface
"NotifyOnCommitAdaptor" to get ack notifications.

--Ari

On Wed, Apr 7, 2010 at 2:51 PM, Kirk True <ki...@mustardgrain.com> wrote:
> Hi Ari,
>
> Question below...
>
> Ariel Rabkin wrote:
>
> If the above is correct, then my next question is - how would I know when I
> can reclaim the files on the local storage? How do I know when they've been
> sent to the collector and thus can be deleted from local storage so as to
> prevent unbounded growth of the files?
>
>
> If you connect to the agent and say "list", you can see how  much data
> from each adaptor has been committed. If it matches the length of your
> file, you can safely deregister the adaptor and delete the file.
>
>
> Using 0.4.0, I never saw -- via the list command -- the file size update.
> The file adapter info would be listed and then simply disappear once the
> file had been transferred. The files are typically less than 1k, so it takes
> very little time to transfer.
>
> What I ended up doing -- which requires no polling from my application --
> was to extend FileAdaptor with a new class named DeleteOnSuccessFileAdaptor
> which after sending the chunk, deletes the file. I think that approaches
> muddles the separation of concerns, but it appears to work correctly (in
> prototype stage).
>
> Thoughts?
>
> Thanks,
> Kirk
>



-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

Re: Embedding Chukwa agent/adapter?

[Kirk True <ki...@mustardgrain.com>]

Hi Ari,

Question below...

Ariel Rabkin wrote:
>> If the above is correct, then my next question is - how would I know when I
>> can reclaim the files on the local storage? How do I know when they've been
>> sent to the collector and thus can be deleted from local storage so as to
>> prevent unbounded growth of the files?
>>     
>
> If you connect to the agent and say "list", you can see how  much data
> from each adaptor has been committed. If it matches the length of your
> file, you can safely deregister the adaptor and delete the file.
>   

Using 0.4.0, I never saw -- via the list command -- the file size 
update. The file adapter info would be listed and then simply disappear 
once the file had been transferred. The files are typically less than 
1k, so it takes very little time to transfer.

What I ended up doing -- which requires no polling from my application 
-- was to extend FileAdaptor with a new class named 
DeleteOnSuccessFileAdaptor which after sending the chunk, deletes the 
file. I think that approaches muddles the separation of concerns, but it 
appears to work correctly (in prototype stage).

Thoughts?

Thanks,
Kirk

Re: Embedding Chukwa agent/adapter?

[Ariel Rabkin <as...@gmail.com>]

On Fri, Apr 2, 2010 at 10:55 AM, Kirk True <ki...@mustardgrain.com> wrote:
> Hi Ari,
>
> Thanks for the info.
>
> So the recommended model is to have my app write out to a directory on local
> storage and on the same machine have an adapter that tails files from that
> same directory and handles the communication to the collector? I can see
> your point about loss of network connectivity - my assumption was that the
> Chukwa adapter buffered/persisted the data internally on network failures.

No.  It wouldn't be too hard to implement store-on-collector-failure
at this point, but it isn't there yet.
There is experimental support in 0.4 and trunk for "pessimistically
write everything to disk."

The preferred model is indeed to write to local files, and then add
adaptors to taste.

> If the above is correct, then my next question is - how would I know when I
> can reclaim the files on the local storage? How do I know when they've been
> sent to the collector and thus can be deleted from local storage so as to
> prevent unbounded growth of the files?

If you connect to the agent and say "list", you can see how  much data
from each adaptor has been committed. If it matches the length of your
file, you can safely deregister the adaptor and delete the file.


-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

Re: Embedding Chukwa agent/adapter?

[Kirk True <ki...@mustardgrain.com>]

Hi Ari,

Thanks for the info.

So the recommended model is to have my app write out to a directory on 
local storage and on the same machine have an adapter that tails files 
from that same directory and handles the communication to the collector? 
I can see your point about loss of network connectivity - my assumption 
was that the Chukwa adapter buffered/persisted the data internally on 
network failures.

If the above is correct, then my next question is - how would I know 
when I can reclaim the files on the local storage? How do I know when 
they've been sent to the collector and thus can be deleted from local 
storage so as to prevent unbounded growth of the files?

Thanks!
Kirk

Ariel Rabkin wrote:
> The benefit of the agent model is that it makes it much easier to do
> failure tolerance. Networks go down, but writes to local disk are
> about as reliable as anything can be. Also, it means that your app
> won't fail, even if you hit a bug in Chukwa, and you won't lose the
> collected data, even if your app crashes.
>
> That said, it is certainly possible to embed the guts of the Chukwa
> adaptors and agent inside your process.  There's two plausible ways I
> see to do it:
>
> Approach 1)
> - Instantiate a Connector, the piece that talks to the collector.
> - The Connector reads from a static queue; you can get ahold of this
> queue via DataFactory.
> - Just push Chunks into the queue, and they'll get sent to the collector.
>
> Approach 2)
> Instantiate a ChukwaAgent, and then call its add(...) method to start adaptors.
>
> Approach 1 is best if all the data you want is inside your
> application; approach 2 is more flexible, but if you go for approach
> two, you might just run Chukwa as a normal external process and be
> done with it.
>
> The backfilling loader (in class BackfillingLoader) may be worth inspecting
>
> --Ari
>
> On Thu, Apr 1, 2010 at 11:33 PM, Kirk True <ki...@mustardgrain.com> wrote:
>   
>> Hi all,
>>
>> We'd like to push data to the collectors straight from our application as it
>> receives data from its clients. In a sense we'd like to embed a Chukwa agent
>> and our adapters inside our web application. Does this make sense or is this
>> approach flawed in some way? Otherwise we'd have to save the data to a file
>> in a shared directory that's then picked up by a stand-alone Chukwa agent
>> via a 'file tailer' or something. Or do we want to use the UDPAdaptor and
>> proxy our data to it, which in turn forwards it to the collector?
>>
>> Thanks,
>> Kirk
>>
>>     
>
>
>
>   

Re: Embedding Chukwa agent/adapter?

[Ariel Rabkin <as...@gmail.com>]

The benefit of the agent model is that it makes it much easier to do
failure tolerance. Networks go down, but writes to local disk are
about as reliable as anything can be. Also, it means that your app
won't fail, even if you hit a bug in Chukwa, and you won't lose the
collected data, even if your app crashes.

That said, it is certainly possible to embed the guts of the Chukwa
adaptors and agent inside your process.  There's two plausible ways I
see to do it:

Approach 1)
- Instantiate a Connector, the piece that talks to the collector.
- The Connector reads from a static queue; you can get ahold of this
queue via DataFactory.
- Just push Chunks into the queue, and they'll get sent to the collector.

Approach 2)
Instantiate a ChukwaAgent, and then call its add(...) method to start adaptors.

Approach 1 is best if all the data you want is inside your
application; approach 2 is more flexible, but if you go for approach
two, you might just run Chukwa as a normal external process and be
done with it.

The backfilling loader (in class BackfillingLoader) may be worth inspecting

--Ari

On Thu, Apr 1, 2010 at 11:33 PM, Kirk True <ki...@mustardgrain.com> wrote:
> Hi all,
>
> We'd like to push data to the collectors straight from our application as it
> receives data from its clients. In a sense we'd like to embed a Chukwa agent
> and our adapters inside our web application. Does this make sense or is this
> approach flawed in some way? Otherwise we'd have to save the data to a file
> in a shared directory that's then picked up by a stand-alone Chukwa agent
> via a 'file tailer' or something. Or do we want to use the UDPAdaptor and
> proxy our data to it, which in turn forwards it to the collector?
>
> Thanks,
> Kirk
>



-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

Re: Embedding Chukwa agent/adapter?

[Jerome Boulon <jb...@netflix.com>]

Hi Kirk,

I've built an agent less solution for Netflix similar in concept with Chukwa
and also compatible at the dataSink level.
It's basically a log4J appender that you have to use in your application
using standard log4 configuration.

I'm using Thrift as the transport protocol and will add Avro support later.
This setup has been benchmark at 120 Millions events/day using 4 collectors
on ec2 (small instances) and the goal is to collect over 600M events/day by
the end of next month.

The code is not yet available but you can read more here:
http://wiki.github.com/jboulon/Honu/ and this is a presentation I did at the
last Facebook User Group:
http://www.slideshare.net/evamtse/hive-user-group-presentation-from-netflix-
3182010-3483386

Send me an email if you want more information since I don't want to hijack
Chukwa ML.

/Jerome.


On 4/1/10 11:33 PM, "Kirk True" <ki...@mustardgrain.com> wrote:

> Hi all,
> 
> We'd like to push data to the collectors straight from our application
> as it receives data from its clients. In a sense we'd like to embed a
> Chukwa agent and our adapters inside our web application. Does this make
> sense or is this approach flawed in some way? Otherwise we'd have to
> save the data to a file in a shared directory that's then picked up by a
> stand-alone Chukwa agent via a 'file tailer' or something. Or do we want
> to use the UDPAdaptor and proxy our data to it, which in turn forwards
> it to the collector?
> 
> Thanks,
> Kirk
>