You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Jaime Solano <jd...@gmail.com> on 2015/06/02 22:47:45 UTC
Joins Benchmark
Hi guys,
Are there benchmarks or numbers showing how Phoenix performs during the
join of two or more huge tables? I'm not familiar with the join
implementation, so I'm not sure if there's a limitation regarding number of
regions, memory, disk, etc.
Any thoughts?
Thanks,
-Jaime
Re: Joins Benchmark
Posted by Vladimir Rodionov <vl...@gmail.com>.
Siva,
If your use case require Phoenix large table joins, you better redesign the
system: keep real-time data in HBase/Phoenix and have a separate analytical
system (Hadoop/Hive) with periodic batch updates from HBase. Its a data
duplication, but using ORCFile/Parquet with compression ON will
reduce your duplication significantly.
-Vlad
On Wed, Jun 3, 2015 at 10:45 AM, Siva <sb...@gmail.com> wrote:
> I agree with you Anil!.
>
> On Tue, Jun 2, 2015 at 9:06 PM, anil gupta <an...@gmail.com> wrote:
>
>> Hi Siva/Jaime,
>>
>> In my opinion:
>> HBase is meant for quick key/value lookup or short range based scans and
>> Hive is meant for Analytical/Datawarehouse kind of workload. Full table
>> scan in HBase is not what HBase is known/popular for. Doing joins is not
>> really a sweet spot for HBase if you are doing full table scans.
>> If you are doing full table scan in HBase then you can also try running a
>> MapReduce job over HBase snapshot. Or You could just use Hive OLAP type
>> workload.
>>
>> Thanks,
>> Anil Gupta
>>
>> On Tue, Jun 2, 2015 at 4:43 PM, Siva <sb...@gmail.com> wrote:
>>
>>> Hi Jaime,
>>>
>>> When we ran queries with complex joins (which involves ~10 tables) on
>>> Phoenix on the tables which has large data, initially we have seen a lot of
>>> issues, queries failed with errors. We started to tune both hbase and
>>> phoenix, now few queries are running fine, but queries with larger data set
>>> still have same issues. Still working on tuning them. The reason for
>>> failures could be because of small cluster, limited by memory and IO.
>>>
>>> On the other hand, same quires with same data size on Hive 14 (with Tez
>>> + ORC format + SNAPPY compression) were finished with in 70~100 seconds. It
>>> would be good if Phoenix can publish the performance results on join
>>> queries.
>>>
>>> Thanks,
>>> Siva.
>>>
>>> On Tue, Jun 2, 2015 at 1:47 PM, Jaime Solano <jd...@gmail.com>
>>> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> Are there benchmarks or numbers showing how Phoenix performs during the
>>>> join of two or more huge tables? I'm not familiar with the join
>>>> implementation, so I'm not sure if there's a limitation regarding number of
>>>> regions, memory, disk, etc.
>>>>
>>>> Any thoughts?
>>>>
>>>> Thanks,
>>>> -Jaime
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>>
>
>
Re: Joins Benchmark
Posted by Siva <sb...@gmail.com>.
I agree with you Anil!.
On Tue, Jun 2, 2015 at 9:06 PM, anil gupta <an...@gmail.com> wrote:
> Hi Siva/Jaime,
>
> In my opinion:
> HBase is meant for quick key/value lookup or short range based scans and
> Hive is meant for Analytical/Datawarehouse kind of workload. Full table
> scan in HBase is not what HBase is known/popular for. Doing joins is not
> really a sweet spot for HBase if you are doing full table scans.
> If you are doing full table scan in HBase then you can also try running a
> MapReduce job over HBase snapshot. Or You could just use Hive OLAP type
> workload.
>
> Thanks,
> Anil Gupta
>
> On Tue, Jun 2, 2015 at 4:43 PM, Siva <sb...@gmail.com> wrote:
>
>> Hi Jaime,
>>
>> When we ran queries with complex joins (which involves ~10 tables) on
>> Phoenix on the tables which has large data, initially we have seen a lot of
>> issues, queries failed with errors. We started to tune both hbase and
>> phoenix, now few queries are running fine, but queries with larger data set
>> still have same issues. Still working on tuning them. The reason for
>> failures could be because of small cluster, limited by memory and IO.
>>
>> On the other hand, same quires with same data size on Hive 14 (with Tez +
>> ORC format + SNAPPY compression) were finished with in 70~100 seconds. It
>> would be good if Phoenix can publish the performance results on join
>> queries.
>>
>> Thanks,
>> Siva.
>>
>> On Tue, Jun 2, 2015 at 1:47 PM, Jaime Solano <jd...@gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> Are there benchmarks or numbers showing how Phoenix performs during the
>>> join of two or more huge tables? I'm not familiar with the join
>>> implementation, so I'm not sure if there's a limitation regarding number of
>>> regions, memory, disk, etc.
>>>
>>> Any thoughts?
>>>
>>> Thanks,
>>> -Jaime
>>>
>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
Re: Joins Benchmark
Posted by anil gupta <an...@gmail.com>.
Hi Siva/Jaime,
In my opinion:
HBase is meant for quick key/value lookup or short range based scans and
Hive is meant for Analytical/Datawarehouse kind of workload. Full table
scan in HBase is not what HBase is known/popular for. Doing joins is not
really a sweet spot for HBase if you are doing full table scans.
If you are doing full table scan in HBase then you can also try running a
MapReduce job over HBase snapshot. Or You could just use Hive OLAP type
workload.
Thanks,
Anil Gupta
On Tue, Jun 2, 2015 at 4:43 PM, Siva <sb...@gmail.com> wrote:
> Hi Jaime,
>
> When we ran queries with complex joins (which involves ~10 tables) on
> Phoenix on the tables which has large data, initially we have seen a lot of
> issues, queries failed with errors. We started to tune both hbase and
> phoenix, now few queries are running fine, but queries with larger data set
> still have same issues. Still working on tuning them. The reason for
> failures could be because of small cluster, limited by memory and IO.
>
> On the other hand, same quires with same data size on Hive 14 (with Tez +
> ORC format + SNAPPY compression) were finished with in 70~100 seconds. It
> would be good if Phoenix can publish the performance results on join
> queries.
>
> Thanks,
> Siva.
>
> On Tue, Jun 2, 2015 at 1:47 PM, Jaime Solano <jd...@gmail.com> wrote:
>
>> Hi guys,
>>
>> Are there benchmarks or numbers showing how Phoenix performs during the
>> join of two or more huge tables? I'm not familiar with the join
>> implementation, so I'm not sure if there's a limitation regarding number of
>> regions, memory, disk, etc.
>>
>> Any thoughts?
>>
>> Thanks,
>> -Jaime
>>
>
>
--
Thanks & Regards,
Anil Gupta
Re: Joins Benchmark
Posted by Siva <sb...@gmail.com>.
Hi Jaime,
When we ran queries with complex joins (which involves ~10 tables) on
Phoenix on the tables which has large data, initially we have seen a lot of
issues, queries failed with errors. We started to tune both hbase and
phoenix, now few queries are running fine, but queries with larger data set
still have same issues. Still working on tuning them. The reason for
failures could be because of small cluster, limited by memory and IO.
On the other hand, same quires with same data size on Hive 14 (with Tez +
ORC format + SNAPPY compression) were finished with in 70~100 seconds. It
would be good if Phoenix can publish the performance results on join
queries.
Thanks,
Siva.
On Tue, Jun 2, 2015 at 1:47 PM, Jaime Solano <jd...@gmail.com> wrote:
> Hi guys,
>
> Are there benchmarks or numbers showing how Phoenix performs during the
> join of two or more huge tables? I'm not familiar with the join
> implementation, so I'm not sure if there's a limitation regarding number of
> regions, memory, disk, etc.
>
> Any thoughts?
>
> Thanks,
> -Jaime
>