You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by alex kamil <al...@gmail.com> on 2014/03/19 03:31:49 UTC
Fwd: native snappy library not available when using count(distinct) grammar
getting the same error with the most recent phoenix v3.0
in order to fix had to add to incubator-phoenix/bin/sqlline.py:
' -Djava.library.path= /var/lib/hadoop/lib/native/Linux-amd64-64'+\
snappy binaries were installed
sudo yum install snappy snappy-devel
ln -sf /usr/lib64/libsnappy.so /var/lib/hadoop/lib/native/Linux-amd64-64/.
ln -sf /usr/lib64/libsnappy.so /var/lib/hbase/lib/native/Linux-amd64-64/.
opened https://issues.apache.org/jira/browse/PHOENIX-877
---------- Forwarded message ----------
From: James Taylor <ja...@apache.org>
Date: Tue, Jan 14, 2014 at 9:55 PM
Subject: Re: native snappy library not available when using count(distinct)
grammar
To: Sean Huo <se...@gmail.com>, Mujtaba Chohan <mc...@salesforce.com>
Cc: Phoenix HBase User <ph...@googlegroups.com>
Thanks for that info, Sean.
Mujtaba - we should add Snappy to the other jar as well. We use it for
compression for larger COUNT DISTINCT and other aggregates as well as to
compress larger ARRAY values.
James
On Tue, Jan 14, 2014 at 6:49 PM, Sean Huo <se...@gmail.com> wrote:
> Also it only happens when the data size of the query is big.
>
>
> On Tuesday, January 14, 2014 4:43:23 PM UTC-8, Sean Huo wrote:
>>
>> I am running on Linux, Mujtaba.
>>
>> On Tuesday, January 14, 2014 4:30:05 PM UTC-8, Mujtaba wrote:
>>>
>>> Hi Sean,
>>>
>>> What is your client platform? Mac or Linux? Snappy is part of client jar
>>> and I'm not getting this exception on my system.
>>>
>>> Thanks,
>>> Mujtaba
>>>
>>> On Tuesday, 14 January 2014 14:43:33 UTC-8, Sean Huo wrote:
>>>>
>>>> So I do find a workaround. add this cmdline argument to the sqline
>>>>
>>>> -Djava.library.path=/usr/lib/hadoop/lib/native
>>>>
>>>> assuming that a snappy implementation is already being installed in
>>>> hadoop.
>>>>
>>>> On Tuesday, January 14, 2014 2:08:37 PM UTC-8, Sean Huo wrote:
>>>>>
>>>>> Any update on this issue? I am encountering the same problem.
>>>>> I am using the trunk build, which is 3.00.snapshot.
>>>>> The jar file does include the snappy library as shown:
>>>>>
>>>>> jar tvf phoenix-3.0.0-SNAPSHOT-client.jar | grep -i snappy
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 META-INF/maven/org.xerial.snappy/
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 META-INF/maven/org.xerial.
>>>>> snappy/snappy-java/
>>>>>
>>>>> 11560 Wed Oct 05 10:34:46 PDT 2011 META-INF/maven/org.xerial.
>>>>> snappy/snappy-java/LICENSE
>>>>>
>>>>> 144 Wed Oct 05 10:34:56 PDT 2011 META-INF/maven/org.xerial.
>>>>> snappy/snappy-java/pom.properties
>>>>>
>>>>> 9273 Wed Oct 05 10:34:48 PDT 2011 META-INF/maven/org.xerial.
>>>>> snappy/snappy-java/pom.xml
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 org/xerial/snappy/
>>>>>
>>>>> 1691 Wed Oct 05 10:34:52 PDT 2011 org/xerial/snappy/OSInfo.class
>>>>>
>>>>> 9444 Wed Oct 05 10:34:52 PDT 2011 org/xerial/snappy/Snappy.class
>>>>>
>>>>> 660 Wed Oct 05 10:34:52 PDT 2011 org/xerial/snappy/
>>>>> SnappyBundleActivator.class
>>>>>
>>>>> 2384 Wed Oct 05 10:34:52 PDT 2011 org/xerial/snappy/SnappyCodec.
>>>>> class
>>>>>
>>>>> 1250 Wed Oct 05 10:34:52 PDT 2011 org/xerial/snappy/SnappyError.
>>>>> class
>>>>>
>>>>> 1790 Wed Oct 05 10:34:52 PDT 2011 org/xerial/snappy/
>>>>> SnappyErrorCode.class
>>>>>
>>>>> 1740 Wed Oct 05 10:34:52 PDT 2011 org/xerial/snappy/
>>>>> SnappyException.class
>>>>>
>>>>> 5354 Wed Oct 05 10:34:52 PDT 2011 org/xerial/snappy/
>>>>> SnappyInputStream.class
>>>>>
>>>>> 11188 Wed Oct 05 10:34:52 PDT 2011 org/xerial/snappy/
>>>>> SnappyLoader.class
>>>>>
>>>>> 1409 Wed Oct 05 10:34:52 PDT 2011 org/xerial/snappy/
>>>>> SnappyNative.class
>>>>>
>>>>> 789 Wed Oct 05 10:34:52 PDT 2011 org/xerial/snappy/
>>>>> SnappyNativeAPI.class
>>>>>
>>>>> 1179 Wed Oct 05 10:34:48 PDT 2011 org/xerial/snappy/
>>>>> SnappyNativeLoader.bytecode
>>>>>
>>>>> 3911 Wed Oct 05 10:34:52 PDT 2011 org/xerial/snappy/
>>>>> SnappyOutputStream.class
>>>>>
>>>>> 17 Wed Oct 05 10:34:48 PDT 2011 org/xerial/snappy/VERSION
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 org/xerial/snappy/native/
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 org/xerial/snappy/native/Linux/
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 org/xerial/snappy/native/
>>>>> Linux/amd64/
>>>>>
>>>>> 991112 Wed Oct 05 10:34:48 PDT 2011 org/xerial/snappy/native/
>>>>> Linux/amd64/libsnappyjava.so
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 org/xerial/snappy/native/
>>>>> Linux/i386/
>>>>>
>>>>> 814468 Wed Oct 05 10:34:48 PDT 2011 org/xerial/snappy/native/
>>>>> Linux/i386/libsnappyjava.so
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 org/xerial/snappy/native/Mac/
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 org/xerial/snappy/native/Mac/i386/
>>>>>
>>>>> 44036 Wed Oct 05 10:34:48 PDT 2011 org/xerial/snappy/native/Mac/
>>>>> i386/libsnappyjava.jnilib
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 org/xerial/snappy/native/Mac/
>>>>> x86_64/
>>>>>
>>>>> 49792 Wed Oct 05 10:34:48 PDT 2011 org/xerial/snappy/native/Mac/
>>>>> x86_64/libsnappyjava.jnilib
>>>>>
>>>>> 71 Wed Oct 05 10:34:48 PDT 2011 org/xerial/snappy/native/README
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 org/xerial/snappy/native/Windows/
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 org/xerial/snappy/native/
>>>>> Windows/amd64/
>>>>>
>>>>> 645632 Wed Oct 05 10:34:48 PDT 2011 org/xerial/snappy/native/
>>>>> Windows/amd64/snappyjava.dll
>>>>>
>>>>> 0 Wed Oct 05 10:34:56 PDT 2011 org/xerial/snappy/native/
>>>>> Windows/x86/
>>>>>
>>>>> 600064 Wed Oct 05 10:34:48 PDT 2011 org/xerial/snappy/native/
>>>>> Windows/x86/snappyjava.dll
>>>>>
>>>>> 0 Wed Oct 03 05:05:16 PDT 2012 org/apache/hadoop/io/compress/
>>>>> snappy/
>>>>>
>>>>> 3960 Wed Oct 03 05:05:16 PDT 2012 org/apache/hadoop/io/compress/
>>>>> SnappyCodec.class
>>>>>
>>>>> 1483 Wed Oct 03 05:05:16 PDT 2012 org/apache/hadoop/io/compress/
>>>>> snappy/LoadSnappy.class
>>>>>
>>>>> 4439 Wed Oct 03 05:05:16 PDT 2012 org/apache/hadoop/io/compress/
>>>>> snappy/SnappyCompressor.class
>>>>>
>>>>> 3976 Wed Oct 03 05:05:16 PDT 2012 org/apache/hadoop/io/compress/
>>>>> snappy/SnappyDecompressor.class
>>>>>
>>>>> It looks like it tries to use org.apache.hadoop.io.compress.snappy.LoadSnappy
>>>>> class to load the library instead of
>>>>>
>>>>> org.xerial.snappy.SnappyLoader.
>>>>>
>>>>> as shown in the stacktrace here
>>>>>
>>>>> java.lang.RuntimeException: native snappy library not available
>>>>>
>>>>> at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(
>>>>> SnappyCodec.java:189)
>>>>>
>>>>> at org.apache.hadoop.io.compress.CodecPool.getDecompressor(
>>>>> CodecPool.java:125)
>>>>>
>>>>> at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.
>>>>> getDecompressor(Compression.java:331)
>>>>>
>>>>> at com.salesforce.phoenix.expression.aggregator.
>>>>> DistinctValueWithCountClientAggregator.aggregate(
>>>>> DistinctValueWithCountClientAggregator.java:70)
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Sean
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thursday, December 19, 2013 8:56:43 AM UTC-8, Mujtaba wrote:
>>>>>>
>>>>>> Correct. Snappy is bundled in Phoenix 2.2.1 client jar (posted on
>>>>>> 12/17) https://github.com/forcedotcom/phoenix/wiki/Download
>>>>>>
>>>>>> //mujtaba
>>>>>>
>>>>>> On Wednesday, 18 December 2013 22:29:17 UTC-8, James Taylor wrote:
>>>>>>>
>>>>>>> The count distinct function requires Snappy when the aggregation
>>>>>>> gets too big. I believe we've added this to the client jar in newer
>>>>>>> releases. Is that correct, Mujtaba?
>>>>>>>
>>>>>>> You can do this yourself too, by adding the Snappy jar to the
>>>>>>> classpath in the sqlline script if you don't want to upgrade.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> James
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Dec 17, 2013 at 4:15 AM, Ming Yang <ya...@gmail.com>wrote:
>>>>>>>
>>>>>>>> create table summary(
>>>>>>>> id integer not null,
>>>>>>>> date date not null
>>>>>>>> device varchar not null
>>>>>>>> vv bigint
>>>>>>>> constraint pk primary key (id, date, device)
>>>>>>>> ) salt_buckets = 30,versions=1,compression='snappy';
>>>>>>>>
>>>>>>>> Loaded 18 million rows,query 1 is ok,query 2 throws out
>>>>>>>> exception.I'm sure that snappy library is available,is this a bug of
>>>>>>>> Phoenix?
>>>>>>>>
>>>>>>>> *query 1* : select id,device,sum(vv) from summary group by
>>>>>>>> id,device;
>>>>>>>>
>>>>>>>> *query 2* : select count(distinct id) from summary;
>>>>>>>> +-------------------------+
>>>>>>>> | DISTINCT_COUNT(ID) |
>>>>>>>> +-------------------------+
>>>>>>>> java.lang.RuntimeException: native snappy library not available
>>>>>>>> at org.apache.hadoop.io.compress.SnappyCodec.
>>>>>>>> getDecompressorType(SnappyCodec.java:189)
>>>>>>>> at org.apache.hadoop.io.compress.CodecPool.getDecompressor(
>>>>>>>> CodecPool.java:125)
>>>>>>>> at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.
>>>>>>>> getDecompressor(Compression.java:331)
>>>>>>>> at com.salesforce.phoenix.expression.aggregator.
>>>>>>>> DistinctValueWithCountClientAggregator.aggregate(
>>>>>>>> DistinctValueWithCountClientAggregator.java:70)
>>>>>>>> at com.salesforce.phoenix.expression.aggregator.
>>>>>>>> ClientAggregators.aggregate(ClientAggregators.java:73)
>>>>>>>> at com.salesforce.phoenix.iterate.
>>>>>>>> GroupedAggregatingResultIterator.next(
>>>>>>>> GroupedAggregatingResultIterator.java:85)
>>>>>>>> at com.salesforce.phoenix.iterate.
>>>>>>>> UngroupedAggregatingResultIterator.next(
>>>>>>>> UngroupedAggregatingResultIterator.java:49)
>>>>>>>> at com.salesforce.phoenix.jdbc.PhoenixResultSet.next(
>>>>>>>> PhoenixResultSet.java:741)
>>>>>>>> at sqlline.SqlLine$IncrementalRows.hasNext(
>>>>>>>> SqlLine.java:2429)
>>>>>>>> at sqlline.SqlLine$TableOutputFormat.print(
>>>>>>>> SqlLine.java:2074)
>>>>>>>> at sqlline.SqlLine.print(SqlLine.java:1735)
>>>>>>>> at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
>>>>>>>> at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
>>>>>>>> at sqlline.SqlLine.dispatch(SqlLine.java:821)
>>>>>>>> at sqlline.SqlLine.begin(SqlLine.java:699)
>>>>>>>> at sqlline.SqlLine.mainWithInputRedirection(
>>>>>>>> SqlLine.java:441)
>>>>>>>> at sqlline.SqlLine.main(SqlLine.java:424)
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Phoenix HBase User" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to phoenix-hbase-user+unsubscribe@googlegroups.com.
>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>>>
>>>>>>>
>>>>>>> --
> You received this message because you are subscribed to the Google Groups
> "Phoenix HBase User" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to phoenix-hbase-user+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
You received this message because you are subscribed to the Google Groups
"Phoenix HBase User" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to phoenix-hbase-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.