async table updates?

[Vishal Kapoor <vi...@gmail.com>]

I have two tables called LIVE and MASTER.
LIVE reports about the MASTER activity and I need to process records in LIVE
almost real time( some business logic)

if I need to store the activity of entities reported by LIVE rows in MASTER
say in ACTIVITY:LAST_REPORTED

I could process my data in LIVE and at the same time update the MASTER's
"LIVE REPORTED" attribute with value of current timestamp.
but I was wondering if I can update the MASTER table in a async call so that
I don't compromise on the speed.
was wondering if anyone tried to achieve this or even if it is advisable...


thanks,
Vishal Kapoor

Re: async table updates?

[Dan Harvey <da...@gmail.com>]

You could use something like the row log library from the Lilly project to queue the processing from one table to the other every time you do a put in HBase. As the queue is in HBase itself too it's all atomic, I'm sure you could get it working with minimal lag between the two tables.

http://www.lilyproject.org/lily/about/playground/hbaserowlog.html

On 23 Feb 2011, at 22:40, Ryan Rawson <ry...@gmail.com> wrote:

> In thrift there is a 'oneway' or 'async' or 'fire and forget' call
> type.  I cant recommend those kinds of approaches, since once your
> system runs into problems you have no feedback.  So if you are asking
> for a one shot, no reply "assume it worked" call, we don't have one
> (nor would I wish that hell on anyone).
> 
> So it boils down to your application architecture.  I'm sure you can
> figure out how to get notifications via callbacks (with background
> threads or threadpools).
> 
> Or did you mean something else?
> -ryan
> 
> On Wed, Feb 23, 2011 at 2:31 PM, Vishal Kapoor
> <vi...@gmail.com> wrote:
>> I have two tables called LIVE and MASTER.
>> LIVE reports about the MASTER activity and I need to process records in LIVE
>> almost real time( some business logic)
>> 
>> if I need to store the activity of entities reported by LIVE rows in MASTER
>> say in ACTIVITY:LAST_REPORTED
>> 
>> I could process my data in LIVE and at the same time update the MASTER's
>> "LIVE REPORTED" attribute with value of current timestamp.
>> but I was wondering if I can update the MASTER table in a async call so that
>> I don't compromise on the speed.
>> was wondering if anyone tried to achieve this or even if it is advisable...
>> 
>> 
>> thanks,
>> Vishal Kapoor
>> 

Re: async table updates?

[Ryan Rawson <ry...@gmail.com>]

In thrift there is a 'oneway' or 'async' or 'fire and forget' call
type.  I cant recommend those kinds of approaches, since once your
system runs into problems you have no feedback.  So if you are asking
for a one shot, no reply "assume it worked" call, we don't have one
(nor would I wish that hell on anyone).

So it boils down to your application architecture.  I'm sure you can
figure out how to get notifications via callbacks (with background
threads or threadpools).

Or did you mean something else?
-ryan

On Wed, Feb 23, 2011 at 2:31 PM, Vishal Kapoor
<vi...@gmail.com> wrote:
> I have two tables called LIVE and MASTER.
> LIVE reports about the MASTER activity and I need to process records in LIVE
> almost real time( some business logic)
>
> if I need to store the activity of entities reported by LIVE rows in MASTER
> say in ACTIVITY:LAST_REPORTED
>
> I could process my data in LIVE and at the same time update the MASTER's
> "LIVE REPORTED" attribute with value of current timestamp.
> but I was wondering if I can update the MASTER table in a async call so that
> I don't compromise on the speed.
> was wondering if anyone tried to achieve this or even if it is advisable...
>
>
> thanks,
> Vishal Kapoor
>