How to stream data out of hbase

[yeshwanth kumar <ye...@gmail.com>]

Hi

i am searching for a way to stream data from hbase,
one way to do is with filters , but i need to query hbase continously,
another way is to read directly from WAL, (i am searching for sample code,
and i found WALReader and WAL.Entry API's.  can i use them directly without
any side effects)

can anyone suggest me a good way to stream data out of hbase, as the write
happens, i want the same data to be pushed to another data source.
please let me know


-Yeshwanth
Can you Imagine what I would do if I could do all I can - Art of War

Re: How to stream data out of hbase

[Rohit Jain <ro...@esgyn.com>]

Besides the fact that it will add overhead to your writes depending on what streaming guarantees you desire (at-most-once, at-least-once, exactly-once), how many subscribers you want to support, the restart capabilities you would want in failure scenarios, etc.

Rohit

On Oct 24, 2017, at 11:44 PM, Josh Elser <el...@apache.org>> wrote:

The most reliably way (read-as, likely to continue working across HBase releases) would probably be to implement a custom ReplicationEndpoint.

This would abstract away the logic behind "tail'ing of WALs" and give you some nicer APIs to leverage. Beware that this would still be a rather significant undertaking that would likely require you to dig into HBase internals to get correct.

On 10/24/17 4:02 PM, yeshwanth kumar wrote:
Hi
i am searching for a way to stream data from hbase,
one way to do is with filters , but i need to query hbase continously,
another way is to read directly from WAL, (i am searching for sample code,
and i found WALReader and WAL.Entry API's.  can i use them directly without
any side effects)
can anyone suggest me a good way to stream data out of hbase, as the write
happens, i want the same data to be pushed to another data source.
please let me know
-Yeshwanth
Can you Imagine what I would do if I could do all I can - Art of War

Re: How to stream data out of hbase

[Josh Elser <el...@apache.org>]

The most reliably way (read-as, likely to continue working across HBase 
releases) would probably be to implement a custom ReplicationEndpoint.

This would abstract away the logic behind "tail'ing of WALs" and give 
you some nicer APIs to leverage. Beware that this would still be a 
rather significant undertaking that would likely require you to dig into 
HBase internals to get correct.

On 10/24/17 4:02 PM, yeshwanth kumar wrote:
> Hi
> 
> i am searching for a way to stream data from hbase,
> one way to do is with filters , but i need to query hbase continously,
> another way is to read directly from WAL, (i am searching for sample code,
> and i found WALReader and WAL.Entry API's.  can i use them directly without
> any side effects)
> 
> can anyone suggest me a good way to stream data out of hbase, as the write
> happens, i want the same data to be pushed to another data source.
> please let me know
> 
> 
> -Yeshwanth
> Can you Imagine what I would do if I could do all I can - Art of War
>