On startup is there a hook to start another framework?

["Hiller, Dean x66079" <de...@broadridge.com>]

I was wondering if there was a way so we could create a function framework so when doing map/reduce over a file in our cluster, we could sort of call

functionSvc.process(incomingActivityDto);

so that we end up processing the dto close to the data especially since the steps are access the account, query table X for the account, query table Y for the account, query table Z for the account, etc. etc....we do a lot of querying on tables and the data related to that account is close by naturally, but we don't want to insert the incomingActivityDto but would rather design an RPC mechanism(JMS, or whatever we want) to invoke.

Then the only question left is how to find out which computer/computers(since nodes are replicated) to send the execution of the function too?


1.       How can I startup a framework in hbase?

2.       How can I locate the primary node the data is on and the secondary nodes?

3.       Also, I would like all function calls for the same account to occur on the primary node or at least the same node if I have a replication factor of 3 so that I can synchronize on activities coming in for the same account


Thanks,
Dean

This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.

Re: On startup is there a hook to start another framework?

[Ted Dunning <td...@maprtech.com>]

At this small size (the 100 or so activities) it is unlikely to make much
sense to have indexes and such.  Simply reading the entire set of activities
into memory, adding new ones and writing out the simplest format possible is
probably as fast as any other implementation.

Indexes are useful to avoid reading unnecessary data from disk.  In this
case, you are going to have all of the information in memory already anyway
so you might as well save the space and avoid the indexes.

On Wed, Jul 13, 2011 at 6:25 AM, Hiller, Dean x66079 <
dean.hiller@broadridge.com> wrote:

> Our main processing is we have 10,000,000 activities come in and each one
> we want to just pass a DTO to the primary node with that account which is
> then responsible for pairing that activity.  We have millions of accounts
> and about 126 activities per account so we would like to have mini-databases
> so to speak in that each account has it's own indexes for activities, and a
> few other tables(around 6) and all these get modified when an activity comes
> into the account.  Any ideas appreciated.
>

RE: On startup is there a hook to start another framework?

["Hiller, Dean x66079" <de...@broadridge.com>]

Hmmmm, how about coprocessors, will the coprocessor always fire on the primary node?

If so, then I am thinking I would just keep writing to the account object itself and the account on it's node would then do all the processing(though I would want this done on that same thread so any error is reported back to the original put)

But then I am not sure I can put/get from other tables in the coprocessor, can I?  I don't need to write to the account at all...just trying to get a trigger to go off and use the data passed through the trigger.

Ideas?

Our main processing is we have 10,000,000 activities come in and each one we want to just pass a DTO to the primary node with that account which is then responsible for pairing that activity.  We have millions of accounts and about 126 activities per account so we would like to have mini-databases so to speak in that each account has it's own indexes for activities, and a few other tables(around 6) and all these get modified when an activity comes into the account.  Any ideas appreciated.


Thanks,
Dean

-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Monday, July 11, 2011 1:16 PM
To: user@hbase.apache.org
Subject: Re: On startup is there a hook to start another framework?

On Mon, Jul 11, 2011 at 8:16 AM, Hiller, Dean  x66079
<de...@broadridge.com> wrote:
> 1.       How can I startup a framework in hbase?
>

Use coprocessors or, subclass it (the latter is not recommended -- it
becomes a pain fast).


> 2.       How can I locate the primary node the data is on and the secondary nodes?
>

For a file?  At HBase level, we don't really have a locality notion.
You'd have to go to HDFS to find the blocks that comprise a file and
where the replicas are located.

> 3.       Also, I would like all function calls for the same account to occur on the primary node or at least the same node if I have a replication factor of 3 so that I can synchronize on activities coming in for the same account
>

I'm not sure how to answer this one.

St.Ack
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.

Re: On startup is there a hook to start another framework?

[Stack <st...@duboce.net>]

On Mon, Jul 11, 2011 at 8:16 AM, Hiller, Dean  x66079
<de...@broadridge.com> wrote:
> 1.       How can I startup a framework in hbase?
>

Use coprocessors or, subclass it (the latter is not recommended -- it
becomes a pain fast).


> 2.       How can I locate the primary node the data is on and the secondary nodes?
>

For a file?  At HBase level, we don't really have a locality notion.
You'd have to go to HDFS to find the blocks that comprise a file and
where the replicas are located.

> 3.       Also, I would like all function calls for the same account to occur on the primary node or at least the same node if I have a replication factor of 3 so that I can synchronize on activities coming in for the same account
>

I'm not sure how to answer this one.

St.Ack