You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by dong wang <el...@gmail.com> on 2015/03/25 10:33:55 UTC
question about hbase-site.xml for KYLIN
Is it necessary for user to do some configuration for hbase-site.xml when
deploy KYLIN?
Re: question about hbase-site.xml for KYLIN
Posted by dong wang <el...@gmail.com>.
thanks hongbin~
2015-03-27 10:01 GMT+08:00 hongbin ma <ma...@apache.org>:
> Can you let me know the reason for number 2? is it because the version
> you're using does not include a "setenv.sh" in bin folder?
>
> If yes, simply adding "export ..." will not work, because in old versions
> the variable KYLIN_LD_LIBRARY_PATH might not get used.
>
> If you can use the latest version on
> http://kylin.incubator.apache.org/download/, please update that.
> Otherwise, you'll need to manually add -Djava.library.path
> "path/to/native/Linux-amd64-64" in kylin.sh's "hbase
> {...} org.apache.hadoop.util.RunJar ${tomcat_root}/bin/bootstrap.jar" as a
> startup parameter.
>
> Kylin will run in a tomcat instance, so we'll need to pass the native
> library path to tomcat via java.library.path.
>
> And please make sure your native lzo libaries are installed in
> path/to/native/Linux-amd64-64
> on your environment
>
>
> On Thu, Mar 26, 2015 at 2:15 PM, dong wang <el...@gmail.com> wrote:
>
> > 1, I configured all the compression properties in core-site.xml, and
> > mapred-site.xml
> >
> > 2, since I cannot find "Before starting Kylin again, you'll need to
> > uncomment KYLIN_LD_LIBRARY_PATH in ${KYLIN_HOME}/bin/setenv.sh", thus, I
> > add "export KYLIN_LD_LIBRARY_PATH" to the very beginning of the file
> > $KYLIN_HOME/bin/kylin.sh
> >
> > 3, as tested, it seems that the HIVE table will use lzo compression,
> > however, the cube table in HBase does't apply lzo compression since when
> > desc the hbase table for the cube, it shows "COMPRESSION => 'NONE'" for
> its
> > detailed information
> >
> > 4, note that when run commands such as "create 'mytable',
> {NAME=>'colfam:',
> > COMPRESSION=>'lzo'} " and "hbase
> > org.apache.hadoop.hbase.util.CompressionTest abc.txt lzo", it works OK
> >
> >
> >
> > 2015-03-26 13:36 GMT+08:00 dong wang <el...@gmail.com>:
> >
> > > as tested, it seems that the HIVE table will use lzo compression, and
> the
> > > cube table in HBase does't apply lzo compression
> > >
> > > 2015-03-26 13:03 GMT+08:00 dong wang <el...@gmail.com>:
> > >
> > >> i c, what's more, if I configure the ALL the compression properties in
> > >> core-site.xml and mapred-site.xml, is it still neccessary for me to
> do
> > the
> > >> change like "export
> > >>
> >
> KYLIN_LD_LIBRARY_PATH=::/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native"
> > >> if I want to use the compression feature?
> > >>
> > >> 2015-03-25 21:17 GMT+08:00 hongbin ma <ma...@apache.org>:
> > >>
> > >>> Usually it's not necessary to change hbase's configuration.
> > >>>
> > >>> However, we do have a place(*conf/kylin_job_conf.xml*) for users to
> > >>> modify
> > >>> MR jobs' configuration. One example would be adding lzo compression
> to
> > >>> Mappers' and Reducers' outputs. *This will reduce the final cube
> size a
> > >>> lot*.
> > >>> Take a look at
> > >>>
> > >>>
> >
> https://github.com/KylinOLAP/Kylin/blob/master/docs/Operations/Advanced%20settings%20of%20Kylin%20environment.md
> > >>> for how to.
> > >>>
> > >>> In a word, as your data size grows, if your cube building is taking
> > >>> significant time or space, you'd better start to think about
> optimizing
> > >>> by
> > >>> adding parameters in kylin_job_conf.xml
> > >>>
> > >>> On Wed, Mar 25, 2015 at 5:33 PM, dong wang <el...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > Is it necessary for user to do some configuration for
> hbase-site.xml
> > >>> when
> > >>> > deploy KYLIN?
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Regards,
> > >>>
> > >>> *Bin Mahone | 马洪宾*
> > >>> Apache Kylin: http://kylin.io
> > >>> Github: https://github.com/binmahone
> > >>>
> > >>
> > >>
> > >
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
Re: question about hbase-site.xml for KYLIN
Posted by hongbin ma <ma...@apache.org>.
Can you let me know the reason for number 2? is it because the version
you're using does not include a "setenv.sh" in bin folder?
If yes, simply adding "export ..." will not work, because in old versions
the variable KYLIN_LD_LIBRARY_PATH might not get used.
If you can use the latest version on
http://kylin.incubator.apache.org/download/, please update that.
Otherwise, you'll need to manually add -Djava.library.path
"path/to/native/Linux-amd64-64" in kylin.sh's "hbase
{...} org.apache.hadoop.util.RunJar ${tomcat_root}/bin/bootstrap.jar" as a
startup parameter.
Kylin will run in a tomcat instance, so we'll need to pass the native
library path to tomcat via java.library.path.
And please make sure your native lzo libaries are installed in
path/to/native/Linux-amd64-64
on your environment
On Thu, Mar 26, 2015 at 2:15 PM, dong wang <el...@gmail.com> wrote:
> 1, I configured all the compression properties in core-site.xml, and
> mapred-site.xml
>
> 2, since I cannot find "Before starting Kylin again, you'll need to
> uncomment KYLIN_LD_LIBRARY_PATH in ${KYLIN_HOME}/bin/setenv.sh", thus, I
> add "export KYLIN_LD_LIBRARY_PATH" to the very beginning of the file
> $KYLIN_HOME/bin/kylin.sh
>
> 3, as tested, it seems that the HIVE table will use lzo compression,
> however, the cube table in HBase does't apply lzo compression since when
> desc the hbase table for the cube, it shows "COMPRESSION => 'NONE'" for its
> detailed information
>
> 4, note that when run commands such as "create 'mytable', {NAME=>'colfam:',
> COMPRESSION=>'lzo'} " and "hbase
> org.apache.hadoop.hbase.util.CompressionTest abc.txt lzo", it works OK
>
>
>
> 2015-03-26 13:36 GMT+08:00 dong wang <el...@gmail.com>:
>
> > as tested, it seems that the HIVE table will use lzo compression, and the
> > cube table in HBase does't apply lzo compression
> >
> > 2015-03-26 13:03 GMT+08:00 dong wang <el...@gmail.com>:
> >
> >> i c, what's more, if I configure the ALL the compression properties in
> >> core-site.xml and mapred-site.xml, is it still neccessary for me to do
> the
> >> change like "export
> >>
> KYLIN_LD_LIBRARY_PATH=::/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native"
> >> if I want to use the compression feature?
> >>
> >> 2015-03-25 21:17 GMT+08:00 hongbin ma <ma...@apache.org>:
> >>
> >>> Usually it's not necessary to change hbase's configuration.
> >>>
> >>> However, we do have a place(*conf/kylin_job_conf.xml*) for users to
> >>> modify
> >>> MR jobs' configuration. One example would be adding lzo compression to
> >>> Mappers' and Reducers' outputs. *This will reduce the final cube size a
> >>> lot*.
> >>> Take a look at
> >>>
> >>>
> https://github.com/KylinOLAP/Kylin/blob/master/docs/Operations/Advanced%20settings%20of%20Kylin%20environment.md
> >>> for how to.
> >>>
> >>> In a word, as your data size grows, if your cube building is taking
> >>> significant time or space, you'd better start to think about optimizing
> >>> by
> >>> adding parameters in kylin_job_conf.xml
> >>>
> >>> On Wed, Mar 25, 2015 at 5:33 PM, dong wang <el...@gmail.com>
> >>> wrote:
> >>>
> >>> > Is it necessary for user to do some configuration for hbase-site.xml
> >>> when
> >>> > deploy KYLIN?
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Regards,
> >>>
> >>> *Bin Mahone | 马洪宾*
> >>> Apache Kylin: http://kylin.io
> >>> Github: https://github.com/binmahone
> >>>
> >>
> >>
> >
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: question about hbase-site.xml for KYLIN
Posted by dong wang <el...@gmail.com>.
1, I configured all the compression properties in core-site.xml, and
mapred-site.xml
2, since I cannot find "Before starting Kylin again, you'll need to
uncomment KYLIN_LD_LIBRARY_PATH in ${KYLIN_HOME}/bin/setenv.sh", thus, I
add "export KYLIN_LD_LIBRARY_PATH" to the very beginning of the file
$KYLIN_HOME/bin/kylin.sh
3, as tested, it seems that the HIVE table will use lzo compression,
however, the cube table in HBase does't apply lzo compression since when
desc the hbase table for the cube, it shows "COMPRESSION => 'NONE'" for its
detailed information
4, note that when run commands such as "create 'mytable', {NAME=>'colfam:',
COMPRESSION=>'lzo'} " and "hbase
org.apache.hadoop.hbase.util.CompressionTest abc.txt lzo", it works OK
2015-03-26 13:36 GMT+08:00 dong wang <el...@gmail.com>:
> as tested, it seems that the HIVE table will use lzo compression, and the
> cube table in HBase does't apply lzo compression
>
> 2015-03-26 13:03 GMT+08:00 dong wang <el...@gmail.com>:
>
>> i c, what's more, if I configure the ALL the compression properties in
>> core-site.xml and mapred-site.xml, is it still neccessary for me to do the
>> change like "export
>> KYLIN_LD_LIBRARY_PATH=::/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native"
>> if I want to use the compression feature?
>>
>> 2015-03-25 21:17 GMT+08:00 hongbin ma <ma...@apache.org>:
>>
>>> Usually it's not necessary to change hbase's configuration.
>>>
>>> However, we do have a place(*conf/kylin_job_conf.xml*) for users to
>>> modify
>>> MR jobs' configuration. One example would be adding lzo compression to
>>> Mappers' and Reducers' outputs. *This will reduce the final cube size a
>>> lot*.
>>> Take a look at
>>>
>>> https://github.com/KylinOLAP/Kylin/blob/master/docs/Operations/Advanced%20settings%20of%20Kylin%20environment.md
>>> for how to.
>>>
>>> In a word, as your data size grows, if your cube building is taking
>>> significant time or space, you'd better start to think about optimizing
>>> by
>>> adding parameters in kylin_job_conf.xml
>>>
>>> On Wed, Mar 25, 2015 at 5:33 PM, dong wang <el...@gmail.com>
>>> wrote:
>>>
>>> > Is it necessary for user to do some configuration for hbase-site.xml
>>> when
>>> > deploy KYLIN?
>>> >
>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> *Bin Mahone | 马洪宾*
>>> Apache Kylin: http://kylin.io
>>> Github: https://github.com/binmahone
>>>
>>
>>
>
Re: question about hbase-site.xml for KYLIN
Posted by dong wang <el...@gmail.com>.
as tested, it seems that the HIVE table will use lzo compression, and the
cube table in HBase does't apply lzo compression
2015-03-26 13:03 GMT+08:00 dong wang <el...@gmail.com>:
> i c, what's more, if I configure the ALL the compression properties in
> core-site.xml and mapred-site.xml, is it still neccessary for me to do the
> change like "export
> KYLIN_LD_LIBRARY_PATH=::/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native"
> if I want to use the compression feature?
>
> 2015-03-25 21:17 GMT+08:00 hongbin ma <ma...@apache.org>:
>
>> Usually it's not necessary to change hbase's configuration.
>>
>> However, we do have a place(*conf/kylin_job_conf.xml*) for users to modify
>> MR jobs' configuration. One example would be adding lzo compression to
>> Mappers' and Reducers' outputs. *This will reduce the final cube size a
>> lot*.
>> Take a look at
>>
>> https://github.com/KylinOLAP/Kylin/blob/master/docs/Operations/Advanced%20settings%20of%20Kylin%20environment.md
>> for how to.
>>
>> In a word, as your data size grows, if your cube building is taking
>> significant time or space, you'd better start to think about optimizing by
>> adding parameters in kylin_job_conf.xml
>>
>> On Wed, Mar 25, 2015 at 5:33 PM, dong wang <el...@gmail.com>
>> wrote:
>>
>> > Is it necessary for user to do some configuration for hbase-site.xml
>> when
>> > deploy KYLIN?
>> >
>>
>>
>>
>> --
>> Regards,
>>
>> *Bin Mahone | 马洪宾*
>> Apache Kylin: http://kylin.io
>> Github: https://github.com/binmahone
>>
>
>
Re: question about hbase-site.xml for KYLIN
Posted by dong wang <el...@gmail.com>.
i c, what's more, if I configure the ALL the compression properties in
core-site.xml and mapred-site.xml, is it still neccessary for me to do the
change like "export
KYLIN_LD_LIBRARY_PATH=::/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native"
if I want to use the compression feature?
2015-03-25 21:17 GMT+08:00 hongbin ma <ma...@apache.org>:
> Usually it's not necessary to change hbase's configuration.
>
> However, we do have a place(*conf/kylin_job_conf.xml*) for users to modify
> MR jobs' configuration. One example would be adding lzo compression to
> Mappers' and Reducers' outputs. *This will reduce the final cube size a
> lot*.
> Take a look at
>
> https://github.com/KylinOLAP/Kylin/blob/master/docs/Operations/Advanced%20settings%20of%20Kylin%20environment.md
> for how to.
>
> In a word, as your data size grows, if your cube building is taking
> significant time or space, you'd better start to think about optimizing by
> adding parameters in kylin_job_conf.xml
>
> On Wed, Mar 25, 2015 at 5:33 PM, dong wang <el...@gmail.com> wrote:
>
> > Is it necessary for user to do some configuration for hbase-site.xml when
> > deploy KYLIN?
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
Re: question about hbase-site.xml for KYLIN
Posted by hongbin ma <ma...@apache.org>.
Usually it's not necessary to change hbase's configuration.
However, we do have a place(*conf/kylin_job_conf.xml*) for users to modify
MR jobs' configuration. One example would be adding lzo compression to
Mappers' and Reducers' outputs. *This will reduce the final cube size a lot*.
Take a look at
https://github.com/KylinOLAP/Kylin/blob/master/docs/Operations/Advanced%20settings%20of%20Kylin%20environment.md
for how to.
In a word, as your data size grows, if your cube building is taking
significant time or space, you'd better start to think about optimizing by
adding parameters in kylin_job_conf.xml
On Wed, Mar 25, 2015 at 5:33 PM, dong wang <el...@gmail.com> wrote:
> Is it necessary for user to do some configuration for hbase-site.xml when
> deploy KYLIN?
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone