Hbase and Chukwa - Question on Agregation

[Vijay <vi...@gmail.com>]

Hi,
Have a question on the Aggregation, I notice the current Chukwa
drive doesn't do any consolidation it just stores all the data into the Date
partitions like month days and weeks...
Consolidation means to me: Adding or averaging the data....

In Hbase we don't need to partition the data because it can hold a large
data set.... but my question is are guys working on Consolidation or
something?

Coz i need that functionality and planning to add those in the post process.

Regards,
</VJ>

Re: Hbase and Chukwa - Question on Agregation

[Eric Yang <ey...@yahoo-inc.com>]

Hi VJ,

The most recent commits in trunk/contrib/chukwa-pig, the framework provides
pig a way to access chukwa records, and our aggregation pig scripts are
committed to trunk/script.

In addition, you will find a new DataLoaderFactory.java in
org.apache.hadoop.chukwa.dataloader.  If you are writing a data loader for
HBASE, extending DataLoaderFactory class should do the job.

Hope this helps

Regards,
Eric

On 6/15/09 10:01 PM, "Vijay" <vi...@gmail.com> wrote:

> Thanks Eric!
> But the thing i don't understand is... we will run pig scripts is it? and
> store the data back to mysql? Where can i learn more abt that?
> 
> Regards,
> </VJ>
> 
> 
> 
> On Mon, Jun 8, 2009 at 2:29 PM, Eric Yang <ey...@yahoo-inc.com> wrote:
> 
>> 

Re: Hbase and Chukwa - Question on Agregation

[Vijay <vi...@gmail.com>]

Thanks Eric!
But the thing i don't understand is... we will run pig scripts is it? and
store the data back to mysql? Where can i learn more abt that?

Regards,
</VJ>



On Mon, Jun 8, 2009 at 2:29 PM, Eric Yang <ey...@yahoo-inc.com> wrote:

>

Re: Hbase and Chukwa - Question on Agregation

[Eric Yang <ey...@yahoo-inc.com>]

Hi Vijay,

There are two consolidation (down sampling) framework in Chukwa.  The first
implementation was done in mysql, where we have a scheduler to periodically
execute sql queries (aggregator.sql) to normalize the data, and rewrite
directly into mysql.  The second implementation is currently under
construction in pig.  There are still some problems (like join) to over come
for the second implementation.

In essence, we would like to have a scheduler to periodically execute a
batch job to normalize the data.  Let's wait and see how the second
implementation turns out by end of this month.  We could implement a direct
map/reduce consolidator after we learn something at a later time.

Regards,
Eric

On 6/8/09 11:39 AM, "Vijay" <vi...@gmail.com> wrote:

> Hi,
> Have a question on the Aggregation, I notice the current Chukwa
> drive doesn't do any consolidation it just stores all the data into the Date
> partitions like month days and weeks...
> Consolidation means to me: Adding or averaging the data....
> 
> In Hbase we don't need to partition the data because it can hold a large
> data set.... but my question is are guys working on Consolidation or
> something?
> 
> Coz i need that functionality and planning to add those in the post process.
> 
> Regards,
> </VJ>