Analysis of Data

[Surindhar <su...@gmail.com>]

Hi,

Does Flume supports Analysis of Data?

Br,

Re: Analysis of Data

[Mike Percy <mp...@apache.org>]

Hi Steven,
Thanks for chiming in! Please see my responses inline:

On Thu, Feb 7, 2013 at 3:04 PM, Steven Yates <sy...@stevendyates.com>wrote:

> The only missing link within the Flume architecture I see in this
> conversation is the actual channel's and brokers themselves which
> orchestrate this lovely undertaking of data collection.
>

Can you define what you mean by channels and brokers in this context? Since
channel is a synonym for queueing event buffer in Flume parlance. Also, can
you elaborate more on what you mean by orchestration? I think I know where
you're going but I don't want to put words in your mouth.

One opportunity I do see (and I may be wrong) is for the data to offloaded
> into a system such as Apache Mahout  before being sent to the sink. Perhaps
> the concept of a ChannelAdapter of sorts? I.e Mahout Adapter ? Just
> thinking out loud and it may be well out of the question.
>

Why not a Mahout sink? Since Mahout often wants sequence files in a
particular format to begin its MapReduce processing (e.g. its k-Means
clustering implementation), Flume is already a good fit with its HDFS sink
and EventSerializers allowing for writing a plugin to format your data
however it needs to go in. In fact that works today if you have a batch
(even 5-minute batch) use case. With today's functionality, you could use
Oozie to coordinate kicking off the Mahout M/R job periodically, as new
data becomes available and the files are rolled.

Perhaps even more interestingly, I can see a use case where you might want
to use Mahout to do streaming / realtime updates driven by Flume in the
form of an interceptor or a Mahout sink. If online machine learning (e.g.
stochastic gradient descent or something else online) was what you were
thinking, I wonder if there are any folks on this list who might have an
interest in helping to work on putting such a thing together.

In any case, I'd like to hear more about specific use cases for streaming
analytics. :)

Regards,
Mike

Re: Analysis of Data

[Mike Percy <mp...@apache.org>]

Thanks for replying Nitin. My thoughts inline:

On Thu, Feb 7, 2013 at 3:22 AM, Nitin Pawar <ni...@gmail.com> wrote:

> 1) Flume is isolated distributed system in the sense one agent does not
> idea about any other agent
>
Avro sinks know about downstream Avro sources, so basically it's a digraph,
right?

2) Flume in the sense when needs to collect data from multiple references
> and work across different data sets, it may not have the entire data set
> needed
>
I see what you are saying, however that is often the case with a streaming
data processing system, right?

3) let us assume we have required data on agents for processing it in
>  batches, do we really want to pressurize a live production server for data
> processing which can be done by systems like storm or hadoop or other
> system?
>
The data can be sent to downstream hops so there is no need to do data
processing on the application tier.

these are my ideas .. i can be totally wrong but just from systems point of
> view it looks good option to keep data acquisition separate from data
> processing and then storing the processed data for further data serving
>
In theory I agree with you, but because Flume can pipe data to downstream
agents who can do the heavy processing, it seems to me that this
requirement is easily fulfilled by Flume.

Regards,
Mike


On Thu, Feb 7, 2013 at 4:29 PM, Mike Percy <mp...@apache.org> wrote:
>
>> Let's take this conversation further. What is missing?
>>
>>
>> On Thu, Feb 7, 2013 at 2:39 AM, Inder Pall <in...@gmail.com> wrote:
>>
>>> flume is a platform to get events to the right sink (HDFS, local-file,
>>> ....)
>>> analytics is not something which falls in it's territory
>>>
>>> - Inder
>>>
>>>
>>> On Thu, Feb 7, 2013 at 3:22 PM, Surindhar <su...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Does Flume supports Analysis of Data?
>>>>
>>>> Br,
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> - Inder
>>> "You are average of the 5 people you spend the most time with"
>>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Analysis of Data

[Steven Yates <sy...@stevendyates.com>]

The only missing link within the Flume architecture I see in this
conversation is the actual channel's and brokers themselves which
orchestrate this lovely undertaking of data collection. One opportunity I do
see (and I may be wrong) is for the data to offloaded into a system such as
Apache Mahout  before being sent to the sink. Perhaps the concept of a
ChannelAdapter of sorts? I.e Mahout Adapter ? Just thinking out loud and it
may be well out of the question.

Thanks
Steve

From:  Nitin Pawar <ni...@gmail.com>
Reply-To:  <us...@flume.apache.org>
Date:  Thu, 7 Feb 2013 16:52:12 +0530
To:  <us...@flume.apache.org>
Subject:  Re: Analysis of Data

1) Flume is isolated distributed system in the sense one agent does not idea
about any other agent
2) Flume in the sense when needs to collect data from multiple references
and work across different data sets, it may not have the entire data set
needed 
3) let us assume we have required data on agents for processing it in
batches, do we really want to pressurize a live production server for data
processing which can be done by systems like storm or hadoop or other
system? 

these are my ideas .. i can be totally wrong but just from systems point of
view it looks good option to keep data acquisition separate from data
processing and then storing the processed data for further data serving



On Thu, Feb 7, 2013 at 4:29 PM, Mike Percy <mp...@apache.org> wrote:
> Let's take this conversation further. What is missing?
> 
> 
> On Thu, Feb 7, 2013 at 2:39 AM, Inder Pall <in...@gmail.com> wrote:
>> flume is a platform to get events to the right sink (HDFS, local-file, ....)
>> analytics is not something which falls in it's territory
>> 
>> - Inder
>> 
>> 
>> On Thu, Feb 7, 2013 at 3:22 PM, Surindhar <su...@gmail.com> wrote:
>>> Hi,
>>> 
>>> Does Flume supports Analysis of Data?
>>> 
>>> Br,
>>> 
>>> 
>> 
>> 
>> 
>> -- 
>> - Inder
>> "You are average of the 5 people you spend the most time with"
> 



-- 
Nitin Pawar

Re: Analysis of Data

[Nitin Pawar <ni...@gmail.com>]

1) Flume is isolated distributed system in the sense one agent does not
idea about any other agent
2) Flume in the sense when needs to collect data from multiple references
and work across different data sets, it may not have the entire data set
needed
3) let us assume we have required data on agents for processing it in
 batches, do we really want to pressurize a live production server for data
processing which can be done by systems like storm or hadoop or other
system?

these are my ideas .. i can be totally wrong but just from systems point of
view it looks good option to keep data acquisition separate from data
processing and then storing the processed data for further data serving



On Thu, Feb 7, 2013 at 4:29 PM, Mike Percy <mp...@apache.org> wrote:

> Let's take this conversation further. What is missing?
>
>
> On Thu, Feb 7, 2013 at 2:39 AM, Inder Pall <in...@gmail.com> wrote:
>
>> flume is a platform to get events to the right sink (HDFS, local-file,
>> ....)
>> analytics is not something which falls in it's territory
>>
>> - Inder
>>
>>
>> On Thu, Feb 7, 2013 at 3:22 PM, Surindhar <su...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Does Flume supports Analysis of Data?
>>>
>>> Br,
>>>
>>>
>>>
>>
>>
>> --
>> - Inder
>> "You are average of the 5 people you spend the most time with"
>>
>
>


-- 
Nitin Pawar

Re: Analysis of Data

[Mike Percy <mp...@apache.org>]

Let's take this conversation further. What is missing?


On Thu, Feb 7, 2013 at 2:39 AM, Inder Pall <in...@gmail.com> wrote:

> flume is a platform to get events to the right sink (HDFS, local-file,
> ....)
> analytics is not something which falls in it's territory
>
> - Inder
>
>
> On Thu, Feb 7, 2013 at 3:22 PM, Surindhar <su...@gmail.com> wrote:
>
>> Hi,
>>
>> Does Flume supports Analysis of Data?
>>
>> Br,
>>
>>
>>
>
>
> --
> - Inder
> "You are average of the 5 people you spend the most time with"
>

Re: Analysis of Data

[Inder Pall <in...@gmail.com>]

flume is a platform to get events to the right sink (HDFS, local-file,
....)
analytics is not something which falls in it's territory

- Inder


On Thu, Feb 7, 2013 at 3:22 PM, Surindhar <su...@gmail.com> wrote:

> Hi,
>
> Does Flume supports Analysis of Data?
>
> Br,
>
>
>


-- 
- Inder
"You are average of the 5 people you spend the most time with"

Re: Analysis of Data

[Bertrand Dechoux <de...@gmail.com>]

>From by point of view, while you could do it, flume is not really meant to
do it. It's not a CEP engine.
But it can be extended. So it really depends on the complexity of your
analysis.

Bertrand

On Thu, Feb 7, 2013 at 11:24 AM, Surindhar <su...@gmail.com> wrote:

> Thanks Nitin,
>
> On Thu, Feb 7, 2013 at 3:45 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> it just supports collection of data
>>
>> it does not understand anything about content of your data
>>
>>
>> On Thu, Feb 7, 2013 at 3:22 PM, Surindhar <su...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Does Flume supports Analysis of Data?
>>>
>>> Br,
>>>
>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>

Re: Analysis of Data

[Surindhar <su...@gmail.com>]

Thanks Nitin,

On Thu, Feb 7, 2013 at 3:45 PM, Nitin Pawar <ni...@gmail.com> wrote:

> it just supports collection of data
>
> it does not understand anything about content of your data
>
>
> On Thu, Feb 7, 2013 at 3:22 PM, Surindhar <su...@gmail.com> wrote:
>
>> Hi,
>>
>> Does Flume supports Analysis of Data?
>>
>> Br,
>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Analysis of Data

[Nitin Pawar <ni...@gmail.com>]

it just supports collection of data

it does not understand anything about content of your data


On Thu, Feb 7, 2013 at 3:22 PM, Surindhar <su...@gmail.com> wrote:

> Hi,
>
> Does Flume supports Analysis of Data?
>
> Br,
>
>
>


-- 
Nitin Pawar