You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Krishna <re...@gmail.com> on 2014/10/02 23:51:22 UTC
Aggregation queries on a big dataset are failing
Hi,
Aggregate queries seem to be working fine on smaller datasets but when the
data needs to be aggregated over millions of rows, query fails with
following error stack. I'm running Phoenix 3.1 on HBase 0.94.18. Any help?
Query is something like this:
> select a.customer_id, a.product_id, count(*) from customer as a join
> product as b on a.product_id = b.product_id where b.category = 'retail'
> group by a.customer_id, a.product_id
Caused by: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.DoNotRetryIOException:
CUSTOMER,\x11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1412194938071.5602bf7e28a72ad6e3db6257b22e38f8.:
com.google.common.hash.BloomFilter.put(Ljava/lang/Object;)Z
at
org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:73)
at
org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:91)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1333)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.internalOpenScanner(HRegionServer.java:2588)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2556)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:354)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1434)
Caused by: java.lang.NoSuchMethodError:
com.google.common.hash.BloomFilter.put(Ljava/lang/Object;)Z
at
org.apache.phoenix.cache.aggcache.SpillMap$MappedByteBufferMap.addElement(SpillMap.java:437)
at org.apache.phoenix.cache.aggcache.SpillMap.put(SpillMap.java:294)
at
org.apache.phoenix.cache.aggcache.SpillManager.spill(SpillManager.java:261)
at
org.apache.phoenix.cache.aggcache.SpillableGroupByCache$1.removeEldestEntry(SpillableGroupByCache.java:190)
at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:431)
at java.util.HashMap.put(HashMap.java:505)
at
org.apache.phoenix.cache.aggcache.SpillableGroupByCache.cache(SpillableGroupByCache.java:249)
at
org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.scanUnordered(GroupedAggregateRegionObserver.java:384)
at
org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.doPostScannerOpen(GroupedAggregateRegionObserver.java:130)
at
org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:89)
... 8 more
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1012)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
at com.sun.proxy.$Proxy6.openScanner(Unknown Source)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:224)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:126)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:42)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:164)
Re: Aggregation queries on a big dataset are failing
Posted by Sean Huo <se...@crunchyroll.com>.
Phoenix pom file has this dependency entry
<guava.version>12.0.1</guava.version>
I would imagine that this particular code path is executed when the
datasize is large. I had the same problem before.
On Thu, Oct 2, 2014 at 4:53 PM, Krishna <re...@gmail.com> wrote:
> Thanks Sean. I'll try that.
> Do you think, this is happening only with large datasets because of the
> spills to disk during "group by" and guava package is used in such
> scenarios?
>
> On Thu, Oct 2, 2014 at 3:31 PM, Sean Huo <se...@crunchyroll.com> wrote:
>
>> You have to upgrade the guava jar on the regionservers. I am using
>> guava-12.0.1.jar.
>>
>> On Thu, Oct 2, 2014 at 2:51 PM, Krishna <re...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Aggregate queries seem to be working fine on smaller datasets but when
>>> the data needs to be aggregated over millions of rows, query fails with
>>> following error stack. I'm running Phoenix 3.1 on HBase 0.94.18. Any help?
>>>
>>> Query is something like this:
>>>
>>>> select a.customer_id, a.product_id, count(*) from customer as a join
>>>> product as b on a.product_id = b.product_id where b.category = 'retail'
>>>> group by a.customer_id, a.product_id
>>>
>>>
>>> Caused by: org.apache.hadoop.ipc.RemoteException:
>>> org.apache.hadoop.hbase.DoNotRetryIOException:
>>> CUSTOMER,\x11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1412194938071.5602bf7e28a72ad6e3db6257b22e38f8.:
>>> com.google.common.hash.BloomFilter.put(Ljava/lang/Object;)Z
>>> at
>>> org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:73)
>>> at
>>> org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:91)
>>> at
>>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1333)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.internalOpenScanner(HRegionServer.java:2588)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2556)
>>> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at
>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:354)
>>> at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1434)
>>> Caused by: java.lang.NoSuchMethodError:
>>> com.google.common.hash.BloomFilter.put(Ljava/lang/Object;)Z
>>> at
>>> org.apache.phoenix.cache.aggcache.SpillMap$MappedByteBufferMap.addElement(SpillMap.java:437)
>>> at
>>> org.apache.phoenix.cache.aggcache.SpillMap.put(SpillMap.java:294)
>>> at
>>> org.apache.phoenix.cache.aggcache.SpillManager.spill(SpillManager.java:261)
>>> at
>>> org.apache.phoenix.cache.aggcache.SpillableGroupByCache$1.removeEldestEntry(SpillableGroupByCache.java:190)
>>> at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:431)
>>> at java.util.HashMap.put(HashMap.java:505)
>>> at
>>> org.apache.phoenix.cache.aggcache.SpillableGroupByCache.cache(SpillableGroupByCache.java:249)
>>> at
>>> org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.scanUnordered(GroupedAggregateRegionObserver.java:384)
>>> at
>>> org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.doPostScannerOpen(GroupedAggregateRegionObserver.java:130)
>>> at
>>> org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:89)
>>> ... 8 more
>>>
>>> at
>>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1012)
>>> at
>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
>>> at com.sun.proxy.$Proxy6.openScanner(Unknown Source)
>>> at
>>> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:224)
>>> at
>>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:126)
>>> at
>>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:42)
>>> at
>>> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:164)
>>>
>>>
>>
>
Re: Aggregation queries on a big dataset are failing
Posted by Krishna <re...@gmail.com>.
Thanks Sean. I'll try that.
Do you think, this is happening only with large datasets because of the
spills to disk during "group by" and guava package is used in such
scenarios?
On Thu, Oct 2, 2014 at 3:31 PM, Sean Huo <se...@crunchyroll.com> wrote:
> You have to upgrade the guava jar on the regionservers. I am using
> guava-12.0.1.jar.
>
> On Thu, Oct 2, 2014 at 2:51 PM, Krishna <re...@gmail.com> wrote:
>
>> Hi,
>>
>> Aggregate queries seem to be working fine on smaller datasets but when
>> the data needs to be aggregated over millions of rows, query fails with
>> following error stack. I'm running Phoenix 3.1 on HBase 0.94.18. Any help?
>>
>> Query is something like this:
>>
>>> select a.customer_id, a.product_id, count(*) from customer as a join
>>> product as b on a.product_id = b.product_id where b.category = 'retail'
>>> group by a.customer_id, a.product_id
>>
>>
>> Caused by: org.apache.hadoop.ipc.RemoteException:
>> org.apache.hadoop.hbase.DoNotRetryIOException:
>> CUSTOMER,\x11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1412194938071.5602bf7e28a72ad6e3db6257b22e38f8.:
>> com.google.common.hash.BloomFilter.put(Ljava/lang/Object;)Z
>> at
>> org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:73)
>> at
>> org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:91)
>> at
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1333)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.internalOpenScanner(HRegionServer.java:2588)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2556)
>> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:354)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1434)
>> Caused by: java.lang.NoSuchMethodError:
>> com.google.common.hash.BloomFilter.put(Ljava/lang/Object;)Z
>> at
>> org.apache.phoenix.cache.aggcache.SpillMap$MappedByteBufferMap.addElement(SpillMap.java:437)
>> at
>> org.apache.phoenix.cache.aggcache.SpillMap.put(SpillMap.java:294)
>> at
>> org.apache.phoenix.cache.aggcache.SpillManager.spill(SpillManager.java:261)
>> at
>> org.apache.phoenix.cache.aggcache.SpillableGroupByCache$1.removeEldestEntry(SpillableGroupByCache.java:190)
>> at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:431)
>> at java.util.HashMap.put(HashMap.java:505)
>> at
>> org.apache.phoenix.cache.aggcache.SpillableGroupByCache.cache(SpillableGroupByCache.java:249)
>> at
>> org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.scanUnordered(GroupedAggregateRegionObserver.java:384)
>> at
>> org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.doPostScannerOpen(GroupedAggregateRegionObserver.java:130)
>> at
>> org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:89)
>> ... 8 more
>>
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1012)
>> at
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
>> at com.sun.proxy.$Proxy6.openScanner(Unknown Source)
>> at
>> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:224)
>> at
>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:126)
>> at
>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:42)
>> at
>> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:164)
>>
>>
>
Re: Aggregation queries on a big dataset are failing
Posted by Sean Huo <se...@crunchyroll.com>.
You have to upgrade the guava jar on the regionservers. I am using
guava-12.0.1.jar.
On Thu, Oct 2, 2014 at 2:51 PM, Krishna <re...@gmail.com> wrote:
> Hi,
>
> Aggregate queries seem to be working fine on smaller datasets but when the
> data needs to be aggregated over millions of rows, query fails with
> following error stack. I'm running Phoenix 3.1 on HBase 0.94.18. Any help?
>
> Query is something like this:
>
>> select a.customer_id, a.product_id, count(*) from customer as a join
>> product as b on a.product_id = b.product_id where b.category = 'retail'
>> group by a.customer_id, a.product_id
>
>
> Caused by: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hbase.DoNotRetryIOException:
> CUSTOMER,\x11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1412194938071.5602bf7e28a72ad6e3db6257b22e38f8.:
> com.google.common.hash.BloomFilter.put(Ljava/lang/Object;)Z
> at
> org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:73)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:91)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1333)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.internalOpenScanner(HRegionServer.java:2588)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2556)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:354)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1434)
> Caused by: java.lang.NoSuchMethodError:
> com.google.common.hash.BloomFilter.put(Ljava/lang/Object;)Z
> at
> org.apache.phoenix.cache.aggcache.SpillMap$MappedByteBufferMap.addElement(SpillMap.java:437)
> at
> org.apache.phoenix.cache.aggcache.SpillMap.put(SpillMap.java:294)
> at
> org.apache.phoenix.cache.aggcache.SpillManager.spill(SpillManager.java:261)
> at
> org.apache.phoenix.cache.aggcache.SpillableGroupByCache$1.removeEldestEntry(SpillableGroupByCache.java:190)
> at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:431)
> at java.util.HashMap.put(HashMap.java:505)
> at
> org.apache.phoenix.cache.aggcache.SpillableGroupByCache.cache(SpillableGroupByCache.java:249)
> at
> org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.scanUnordered(GroupedAggregateRegionObserver.java:384)
> at
> org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.doPostScannerOpen(GroupedAggregateRegionObserver.java:130)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:89)
> ... 8 more
>
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1012)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
> at com.sun.proxy.$Proxy6.openScanner(Unknown Source)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:224)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:126)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:42)
> at
> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:164)
>
>