You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by hongbin ma <ma...@apache.org> on 2016/09/03 09:36:41 UTC

Re: Turn off some hive configuration commands when they are not allowed

do we have any progress on such documents?

On Tue, Aug 2, 2016 at 8:50 AM, Yiming Liu <li...@gmail.com> wrote:

> Thanks, Shaofeng. It makes sense to grant enough privileges to Kylin for
> Cube building. Just in some extreme cases, the privilege issue will be a
> show stop.
>
> The privilege document is great. It's very helpful for Hadoop system
> administrator.
>
> 2016-08-01 9:38 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>
> > Hi Yiming,
> >
> > The "mapreduce.job.reduces"  need by set at runtime, whose number is
> > calculated based on user tables' size, it couldn't be pre-configured.
> >
> > The "hive.merge.mapredfiles=false" can be externalized to the conf file;
> > The hive merge is not needed since 1.5.3, I set in code to ensure it will
> > be not be enabled (config files before 1.5.3 has this param set to true).
> >
> > For other parameters, I think they're optional, but it is better to keep
> as
> > they're good for performance, like dfs.replication=2, compress.codec etc.
> >
> > Usually in a hadoop cluster, Apache Kylin should be treated as a
> > priviledged user (instead of a normal user like analyst), which can
> execute
> > necessary hadoop/hdfs/hbase/hive actions (like mkdir, create htable,
> etc);
> > To achieve this, the administartor need do some configurations and
> > authorizations; What we need do is to compose a document to list
> > these privileges, what's your opinion?
> >
> > Thanks for the comment!
> >
> >
> > 2016-07-30 14:03 GMT+08:00 Yiming Liu <li...@gmail.com>:
> >
> > > Hi Kylin dev,
> > >
> > > The first step is building cube is to CreateFlatHiveTable, it will
> call a
> > > few hive configuration commands, such as
> > > CreateFlatHiveTableStep line 78 and 79.
> > > set mapreduce.job.reduces=numReduces
> > > set hive.merge.mapredfiles=false
> > >
> > > Are these commands necessary for the cube building? Could we configure
> > them
> > > in files? I met some cases, where the hiveserver would say
> "Configuration
> > > is not allowed to modify at runtime". It will break the build.
> > >
> > > Maybe there are some other hard code hadoop commands still. It will be
> > more
> > > friendly if they could turn off on demand.
> > >
> > > --
> > > With Warm regards
> > >
> > > Yiming Liu (刘一鸣)
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi
> >
>
>
>
> --
> With Warm regards
>
> Yiming Liu (刘一鸣)
>



-- 
Regards,

*Bin Mahone | 马洪宾*

Re: Turn off some hive configuration commands when they are not allowed

Posted by Yiming Liu <li...@gmail.com>.
https://issues.apache.org/jira/browse/KYLIN-1938 for tracking this issue.

2016-09-03 22:53 GMT+08:00 hongbin ma <ma...@apache.org>:

> Do we have JIRAs to track the issue? it's easy to forget
>
> On Sat, Sep 3, 2016 at 5:57 PM, Yiming Liu <li...@gmail.com>
> wrote:
>
> > No progress, pending.
> >
> > 2016-09-03 17:36 GMT+08:00 hongbin ma <ma...@apache.org>:
> >
> > > do we have any progress on such documents?
> > >
> > > On Tue, Aug 2, 2016 at 8:50 AM, Yiming Liu <li...@gmail.com>
> > > wrote:
> > >
> > > > Thanks, Shaofeng. It makes sense to grant enough privileges to Kylin
> > for
> > > > Cube building. Just in some extreme cases, the privilege issue will
> be
> > a
> > > > show stop.
> > > >
> > > > The privilege document is great. It's very helpful for Hadoop system
> > > > administrator.
> > > >
> > > > 2016-08-01 9:38 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> > > >
> > > > > Hi Yiming,
> > > > >
> > > > > The "mapreduce.job.reduces"  need by set at runtime, whose number
> is
> > > > > calculated based on user tables' size, it couldn't be
> pre-configured.
> > > > >
> > > > > The "hive.merge.mapredfiles=false" can be externalized to the conf
> > > file;
> > > > > The hive merge is not needed since 1.5.3, I set in code to ensure
> it
> > > will
> > > > > be not be enabled (config files before 1.5.3 has this param set to
> > > true).
> > > > >
> > > > > For other parameters, I think they're optional, but it is better to
> > > keep
> > > > as
> > > > > they're good for performance, like dfs.replication=2,
> compress.codec
> > > etc.
> > > > >
> > > > > Usually in a hadoop cluster, Apache Kylin should be treated as a
> > > > > priviledged user (instead of a normal user like analyst), which can
> > > > execute
> > > > > necessary hadoop/hdfs/hbase/hive actions (like mkdir, create
> htable,
> > > > etc);
> > > > > To achieve this, the administartor need do some configurations and
> > > > > authorizations; What we need do is to compose a document to list
> > > > > these privileges, what's your opinion?
> > > > >
> > > > > Thanks for the comment!
> > > > >
> > > > >
> > > > > 2016-07-30 14:03 GMT+08:00 Yiming Liu <li...@gmail.com>:
> > > > >
> > > > > > Hi Kylin dev,
> > > > > >
> > > > > > The first step is building cube is to CreateFlatHiveTable, it
> will
> > > > call a
> > > > > > few hive configuration commands, such as
> > > > > > CreateFlatHiveTableStep line 78 and 79.
> > > > > > set mapreduce.job.reduces=numReduces
> > > > > > set hive.merge.mapredfiles=false
> > > > > >
> > > > > > Are these commands necessary for the cube building? Could we
> > > configure
> > > > > them
> > > > > > in files? I met some cases, where the hiveserver would say
> > > > "Configuration
> > > > > > is not allowed to modify at runtime". It will break the build.
> > > > > >
> > > > > > Maybe there are some other hard code hadoop commands still. It
> will
> > > be
> > > > > more
> > > > > > friendly if they could turn off on demand.
> > > > > >
> > > > > > --
> > > > > > With Warm regards
> > > > > >
> > > > > > Yiming Liu (刘一鸣)
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > > Shaofeng Shi
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > With Warm regards
> > > >
> > > > Yiming Liu (刘一鸣)
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > *Bin Mahone | 马洪宾*
> > >
> >
> >
> >
> > --
> > With Warm regards
> >
> > Yiming Liu (刘一鸣)
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
>



-- 
With Warm regards

Yiming Liu (刘一鸣)

Re: Turn off some hive configuration commands when they are not allowed

Posted by hongbin ma <ma...@apache.org>.
Do we have JIRAs to track the issue? it's easy to forget

On Sat, Sep 3, 2016 at 5:57 PM, Yiming Liu <li...@gmail.com> wrote:

> No progress, pending.
>
> 2016-09-03 17:36 GMT+08:00 hongbin ma <ma...@apache.org>:
>
> > do we have any progress on such documents?
> >
> > On Tue, Aug 2, 2016 at 8:50 AM, Yiming Liu <li...@gmail.com>
> > wrote:
> >
> > > Thanks, Shaofeng. It makes sense to grant enough privileges to Kylin
> for
> > > Cube building. Just in some extreme cases, the privilege issue will be
> a
> > > show stop.
> > >
> > > The privilege document is great. It's very helpful for Hadoop system
> > > administrator.
> > >
> > > 2016-08-01 9:38 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> > >
> > > > Hi Yiming,
> > > >
> > > > The "mapreduce.job.reduces"  need by set at runtime, whose number is
> > > > calculated based on user tables' size, it couldn't be pre-configured.
> > > >
> > > > The "hive.merge.mapredfiles=false" can be externalized to the conf
> > file;
> > > > The hive merge is not needed since 1.5.3, I set in code to ensure it
> > will
> > > > be not be enabled (config files before 1.5.3 has this param set to
> > true).
> > > >
> > > > For other parameters, I think they're optional, but it is better to
> > keep
> > > as
> > > > they're good for performance, like dfs.replication=2, compress.codec
> > etc.
> > > >
> > > > Usually in a hadoop cluster, Apache Kylin should be treated as a
> > > > priviledged user (instead of a normal user like analyst), which can
> > > execute
> > > > necessary hadoop/hdfs/hbase/hive actions (like mkdir, create htable,
> > > etc);
> > > > To achieve this, the administartor need do some configurations and
> > > > authorizations; What we need do is to compose a document to list
> > > > these privileges, what's your opinion?
> > > >
> > > > Thanks for the comment!
> > > >
> > > >
> > > > 2016-07-30 14:03 GMT+08:00 Yiming Liu <li...@gmail.com>:
> > > >
> > > > > Hi Kylin dev,
> > > > >
> > > > > The first step is building cube is to CreateFlatHiveTable, it will
> > > call a
> > > > > few hive configuration commands, such as
> > > > > CreateFlatHiveTableStep line 78 and 79.
> > > > > set mapreduce.job.reduces=numReduces
> > > > > set hive.merge.mapredfiles=false
> > > > >
> > > > > Are these commands necessary for the cube building? Could we
> > configure
> > > > them
> > > > > in files? I met some cases, where the hiveserver would say
> > > "Configuration
> > > > > is not allowed to modify at runtime". It will break the build.
> > > > >
> > > > > Maybe there are some other hard code hadoop commands still. It will
> > be
> > > > more
> > > > > friendly if they could turn off on demand.
> > > > >
> > > > > --
> > > > > With Warm regards
> > > > >
> > > > > Yiming Liu (刘一鸣)
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > > Shaofeng Shi
> > > >
> > >
> > >
> > >
> > > --
> > > With Warm regards
> > >
> > > Yiming Liu (刘一鸣)
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> >
>
>
>
> --
> With Warm regards
>
> Yiming Liu (刘一鸣)
>



-- 
Regards,

*Bin Mahone | 马洪宾*

Re: Turn off some hive configuration commands when they are not allowed

Posted by Yiming Liu <li...@gmail.com>.
No progress, pending.

2016-09-03 17:36 GMT+08:00 hongbin ma <ma...@apache.org>:

> do we have any progress on such documents?
>
> On Tue, Aug 2, 2016 at 8:50 AM, Yiming Liu <li...@gmail.com>
> wrote:
>
> > Thanks, Shaofeng. It makes sense to grant enough privileges to Kylin for
> > Cube building. Just in some extreme cases, the privilege issue will be a
> > show stop.
> >
> > The privilege document is great. It's very helpful for Hadoop system
> > administrator.
> >
> > 2016-08-01 9:38 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> >
> > > Hi Yiming,
> > >
> > > The "mapreduce.job.reduces"  need by set at runtime, whose number is
> > > calculated based on user tables' size, it couldn't be pre-configured.
> > >
> > > The "hive.merge.mapredfiles=false" can be externalized to the conf
> file;
> > > The hive merge is not needed since 1.5.3, I set in code to ensure it
> will
> > > be not be enabled (config files before 1.5.3 has this param set to
> true).
> > >
> > > For other parameters, I think they're optional, but it is better to
> keep
> > as
> > > they're good for performance, like dfs.replication=2, compress.codec
> etc.
> > >
> > > Usually in a hadoop cluster, Apache Kylin should be treated as a
> > > priviledged user (instead of a normal user like analyst), which can
> > execute
> > > necessary hadoop/hdfs/hbase/hive actions (like mkdir, create htable,
> > etc);
> > > To achieve this, the administartor need do some configurations and
> > > authorizations; What we need do is to compose a document to list
> > > these privileges, what's your opinion?
> > >
> > > Thanks for the comment!
> > >
> > >
> > > 2016-07-30 14:03 GMT+08:00 Yiming Liu <li...@gmail.com>:
> > >
> > > > Hi Kylin dev,
> > > >
> > > > The first step is building cube is to CreateFlatHiveTable, it will
> > call a
> > > > few hive configuration commands, such as
> > > > CreateFlatHiveTableStep line 78 and 79.
> > > > set mapreduce.job.reduces=numReduces
> > > > set hive.merge.mapredfiles=false
> > > >
> > > > Are these commands necessary for the cube building? Could we
> configure
> > > them
> > > > in files? I met some cases, where the hiveserver would say
> > "Configuration
> > > > is not allowed to modify at runtime". It will break the build.
> > > >
> > > > Maybe there are some other hard code hadoop commands still. It will
> be
> > > more
> > > > friendly if they could turn off on demand.
> > > >
> > > > --
> > > > With Warm regards
> > > >
> > > > Yiming Liu (刘一鸣)
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > Shaofeng Shi
> > >
> >
> >
> >
> > --
> > With Warm regards
> >
> > Yiming Liu (刘一鸣)
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
>



-- 
With Warm regards

Yiming Liu (刘一鸣)