Re: real time Query engine Spark-SQL on Hbase

[Corey Nolet <cj...@gmail.com>]

A tad off topic, but could still be relevant.

Accumulo's design is a tad different in the realm of being able to shard
and perform set intersections/unions server-side (through seeks). I've got
an adapter for Spark SQL on top of a document store implementation in
Accumulo that accepts the push-down predicates and actually performs query
on the tablet servers. This strategy may be useful to you [1].

[1]
https://github.com/calrissian/accumulo-recipes/blob/master/thirdparty/spark/src/test/scala/org/calrissian/accumulorecipes/spark/sql/EventStoreFilteredTest.scala

On Thu, Apr 30, 2015 at 10:54 AM, Ted Yu <yu...@gmail.com> wrote:

> bq. a single query on one filter criteria
>
> Can you tell us more about your filter ? How selective is it ?
>
> Which hbase release are you using ?
>
> Cheers
>
> On Thu, Apr 30, 2015 at 7:23 AM, Siddharth Ubale <
> siddharth.ubale@syncoms.com> wrote:
>
>>  Hi,
>>
>>
>>
>> I want to use Spark as Query engine on HBase with sub second latency.
>>
>>
>>
>> I am  using Spark 1.3  version. And followed the steps below on Hbase
>> table with around 3.5 lac rows :
>>
>>
>>
>> *1.       *Mapped the Dataframe to Hbase table .RDDCustomers maps to the
>> hbase table which is used to create the Dataframe.
>>
>> *“ DataFrame schemaCustomers = sqlInstance*
>>
>> *
>> .createDataFrame(SparkContextImpl.getRddCustomers(),*
>>
>> *
>> Customers.class);” *
>>
>> 2.       Used registertemp table i.e”
>> *schemaCustomers.registerTempTable("customers");”*
>>
>> 3.       Running the query on Dataframe using Sqlcontext Instance.
>>
>>
>>
>> What I am observing is that for a single query on one filter criteria the
>> query is taking 7-8 seconds? And the time increases as I am increasing the
>> number of rows in Hbase table. Also, there was one time when I was getting
>> query response under 1-2 seconds. Seems like strange behavior.
>>
>> Is this expected behavior from Spark or am I missing something here?
>>
>> Can somebody help me understand this scenario . Please assist.
>>
>>
>>
>> Thanks,
>>
>> Siddharth Ubale,
>>
>>
>>
>
>