You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Mars J <xu...@gmail.com> on 2016/02/23 10:54:11 UTC
kylin's compression problem when I comment the related items
Hello,
I ran into a problem as the subject. Details as follow:
version of my envr
OS , CentOS 7
Apache Hadoop 2.7.2
Apache Hive 1.2.1
Apache HBase 1.1.3
Apache Kylin 1.2
I ran the ${KYLIN_HOME}/bin/sample.sh and build it,but the build progress
is Error.The log print error caused by RuntimeException: native snappy
library not available: this version of libhadoop was built without snappy
support.
I have already edit the conf/kylin.properties to set value to 'false' even
delete all items which include compress,but the error log also shows the
information below:(at the same time, I have ignore the related items in
$HADOOP_HOME/etc/hadoop/mapred-site.xml)
SET mapreduce.job.split.metainfo.maxsize=-1;
SET mapred.compress.map.output=true;
SET mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET mapred.output.compress=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET mapred.output.compression.type=BLOCK;
SET mapreduce.job.max.split.locations=2000;
SET dfs.replication=2;
SET hive.merge.mapfiles=true;
SET hive.merge.mapredfiles=true;
SET hive.merge.size.per.task=268435456;
SET hive.support.concurrency=false;
SET hive.exec.compress.output=true;
SET hive.auto.convert.join.noconditionaltask = true;
SET hive.auto.convert.join.noconditionaltask.size = 300000000;
Why it execute the set statement after I ignore the compress items like
mapred.map.output.compress etc.
Is that some configurations I havn't done yet?
Re: kylin's compression problem when I comment the related items
Posted by Li Yang <li...@apache.org>.
Just looked at the software versions. Please get the "Binary Package (for
running on HBase 1.1.3 or above)" from http://kylin.apache.org/download/
The standard Kylin 1.2 does not run on HBase 1.x
On Wed, Feb 24, 2016 at 10:14 AM, yu feng <ol...@gmail.com> wrote:
> you should check some file like 'conf/kylin_job_conf.xml', those 'set key
> = value' read from this file.
>
> 2016-02-23 17:54 GMT+08:00 Mars J <xu...@gmail.com>:
>
>> Hello,
>> I ran into a problem as the subject. Details as follow:
>>
>> version of my envr
>> OS , CentOS 7
>> Apache Hadoop 2.7.2
>> Apache Hive 1.2.1
>> Apache HBase 1.1.3
>> Apache Kylin 1.2
>>
>> I ran the ${KYLIN_HOME}/bin/sample.sh and build it,but the build
>> progress is Error.The log print error caused by RuntimeException: native
>> snappy library not available: this version of libhadoop was built without
>> snappy support.
>>
>> I have already edit the conf/kylin.properties to set value to 'false'
>> even delete all items which include compress,but the error log also shows
>> the information below:(at the same time, I have ignore the related items in
>> $HADOOP_HOME/etc/hadoop/mapred-site.xml)
>>
>> SET mapreduce.job.split.metainfo.maxsize=-1;
>> SET mapred.compress.map.output=true;
>> SET mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
>> SET mapred.output.compress=true;
>> SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
>> SET mapred.output.compression.type=BLOCK;
>> SET mapreduce.job.max.split.locations=2000;
>> SET dfs.replication=2;
>> SET hive.merge.mapfiles=true;
>> SET hive.merge.mapredfiles=true;
>> SET hive.merge.size.per.task=268435456;
>> SET hive.support.concurrency=false;
>> SET hive.exec.compress.output=true;
>> SET hive.auto.convert.join.noconditionaltask = true;
>> SET hive.auto.convert.join.noconditionaltask.size = 300000000;
>>
>> Why it execute the set statement after I ignore the compress items like
>> mapred.map.output.compress etc.
>>
>> Is that some configurations I havn't done yet?
>>
>
>
Re: kylin's compression problem when I comment the related items
Posted by yu feng <ol...@gmail.com>.
you should check some file like 'conf/kylin_job_conf.xml', those 'set key =
value' read from this file.
2016-02-23 17:54 GMT+08:00 Mars J <xu...@gmail.com>:
> Hello,
> I ran into a problem as the subject. Details as follow:
>
> version of my envr
> OS , CentOS 7
> Apache Hadoop 2.7.2
> Apache Hive 1.2.1
> Apache HBase 1.1.3
> Apache Kylin 1.2
>
> I ran the ${KYLIN_HOME}/bin/sample.sh and build it,but the build progress
> is Error.The log print error caused by RuntimeException: native snappy
> library not available: this version of libhadoop was built without snappy
> support.
>
> I have already edit the conf/kylin.properties to set value to 'false' even
> delete all items which include compress,but the error log also shows the
> information below:(at the same time, I have ignore the related items in
> $HADOOP_HOME/etc/hadoop/mapred-site.xml)
>
> SET mapreduce.job.split.metainfo.maxsize=-1;
> SET mapred.compress.map.output=true;
> SET mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET mapred.output.compress=true;
> SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET mapred.output.compression.type=BLOCK;
> SET mapreduce.job.max.split.locations=2000;
> SET dfs.replication=2;
> SET hive.merge.mapfiles=true;
> SET hive.merge.mapredfiles=true;
> SET hive.merge.size.per.task=268435456;
> SET hive.support.concurrency=false;
> SET hive.exec.compress.output=true;
> SET hive.auto.convert.join.noconditionaltask = true;
> SET hive.auto.convert.join.noconditionaltask.size = 300000000;
>
> Why it execute the set statement after I ignore the compress items like
> mapred.map.output.compress etc.
>
> Is that some configurations I havn't done yet?
>