Fwd: Parallel scan on regions only

[Davide Spataro <da...@gmail.com>]

Hi everyone,

I recently joined a project that uses hbase to store large amount of
granular material simulation data (e.g.  thousands of particles's position
and velocity over time). What I need to do is to efficiently gather the
data from Hbase and perform parallel rendering of them.

The parallel rendering framework I'm using is written in C/C++ and uses MPI
to coordinate the parallel job (ICET library). Each rendering process
should fetch a portion of data from hbase independently and then coordinate
to perform the final rendered image.

What I would like to do is to perform the same scan operation on all
regions in parallel and fetch their result to the corresponding renderer.

I'm trying to implement this mechanism using endpoint coprocessors .
Basically each coprocessor receives the same scan query, performs it
locally (at  region level) and send data  to a specific  rendering process
over network (using sockets).

Do you have guys  any better advice on how  to implement this in hbase?

Thanks,

Davide

P.S. I'm running hbase 1.2 on top of hadoop 2.7

Re: Parallel scan on regions only

[Stack <st...@duboce.net>]

On Tue, Apr 12, 2016 at 3:15 AM, Davide Spataro <da...@gmail.com>
wrote:

> ...
> What I would like to do is to perform the same scan operation on all
> regions in parallel and fetch their result to the corresponding renderer.
>


> I'm trying to implement this mechanism using endpoint coprocessors .
> Basically each coprocessor receives the same scan query, performs it
> locally (at  region level) and send data  to a specific  rendering process
> over network (using sockets).
>
> Do you have guys  any better advice on how  to implement this in hbase?
>
>

Sounds like Admin#getTableRegions(TableName) which returns info on each
region in a table and then setting up a Scan per Region is not what you
want; you are trying to save a hop by having writing your rendering app
rather than return to the client and then to the rendering process?

St.Ack



> Thanks,
>
> Davide
>
> P.S. I'm running hbase 1.2 on top of hadoop 2.7
>