You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Yiming Liu <li...@gmail.com> on 2016/07/30 06:03:13 UTC
Turn off some hive configuration commands when they are not allowed
Hi Kylin dev,
The first step is building cube is to CreateFlatHiveTable, it will call a
few hive configuration commands, such as
CreateFlatHiveTableStep line 78 and 79.
set mapreduce.job.reduces=numReduces
set hive.merge.mapredfiles=false
Are these commands necessary for the cube building? Could we configure them
in files? I met some cases, where the hiveserver would say "Configuration
is not allowed to modify at runtime". It will break the build.
Maybe there are some other hard code hadoop commands still. It will be more
friendly if they could turn off on demand.
--
With Warm regards
Yiming Liu (刘一鸣)
Re: Turn off some hive configuration commands when they are not allowed
Posted by Yiming Liu <li...@gmail.com>.
https://issues.apache.org/jira/browse/KYLIN-1938 for tracking this issue.
2016-09-03 22:53 GMT+08:00 hongbin ma <ma...@apache.org>:
> Do we have JIRAs to track the issue? it's easy to forget
>
> On Sat, Sep 3, 2016 at 5:57 PM, Yiming Liu <li...@gmail.com>
> wrote:
>
> > No progress, pending.
> >
> > 2016-09-03 17:36 GMT+08:00 hongbin ma <ma...@apache.org>:
> >
> > > do we have any progress on such documents?
> > >
> > > On Tue, Aug 2, 2016 at 8:50 AM, Yiming Liu <li...@gmail.com>
> > > wrote:
> > >
> > > > Thanks, Shaofeng. It makes sense to grant enough privileges to Kylin
> > for
> > > > Cube building. Just in some extreme cases, the privilege issue will
> be
> > a
> > > > show stop.
> > > >
> > > > The privilege document is great. It's very helpful for Hadoop system
> > > > administrator.
> > > >
> > > > 2016-08-01 9:38 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> > > >
> > > > > Hi Yiming,
> > > > >
> > > > > The "mapreduce.job.reduces" need by set at runtime, whose number
> is
> > > > > calculated based on user tables' size, it couldn't be
> pre-configured.
> > > > >
> > > > > The "hive.merge.mapredfiles=false" can be externalized to the conf
> > > file;
> > > > > The hive merge is not needed since 1.5.3, I set in code to ensure
> it
> > > will
> > > > > be not be enabled (config files before 1.5.3 has this param set to
> > > true).
> > > > >
> > > > > For other parameters, I think they're optional, but it is better to
> > > keep
> > > > as
> > > > > they're good for performance, like dfs.replication=2,
> compress.codec
> > > etc.
> > > > >
> > > > > Usually in a hadoop cluster, Apache Kylin should be treated as a
> > > > > priviledged user (instead of a normal user like analyst), which can
> > > > execute
> > > > > necessary hadoop/hdfs/hbase/hive actions (like mkdir, create
> htable,
> > > > etc);
> > > > > To achieve this, the administartor need do some configurations and
> > > > > authorizations; What we need do is to compose a document to list
> > > > > these privileges, what's your opinion?
> > > > >
> > > > > Thanks for the comment!
> > > > >
> > > > >
> > > > > 2016-07-30 14:03 GMT+08:00 Yiming Liu <li...@gmail.com>:
> > > > >
> > > > > > Hi Kylin dev,
> > > > > >
> > > > > > The first step is building cube is to CreateFlatHiveTable, it
> will
> > > > call a
> > > > > > few hive configuration commands, such as
> > > > > > CreateFlatHiveTableStep line 78 and 79.
> > > > > > set mapreduce.job.reduces=numReduces
> > > > > > set hive.merge.mapredfiles=false
> > > > > >
> > > > > > Are these commands necessary for the cube building? Could we
> > > configure
> > > > > them
> > > > > > in files? I met some cases, where the hiveserver would say
> > > > "Configuration
> > > > > > is not allowed to modify at runtime". It will break the build.
> > > > > >
> > > > > > Maybe there are some other hard code hadoop commands still. It
> will
> > > be
> > > > > more
> > > > > > friendly if they could turn off on demand.
> > > > > >
> > > > > > --
> > > > > > With Warm regards
> > > > > >
> > > > > > Yiming Liu (刘一鸣)
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > > Shaofeng Shi
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > With Warm regards
> > > >
> > > > Yiming Liu (刘一鸣)
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > *Bin Mahone | 马洪宾*
> > >
> >
> >
> >
> > --
> > With Warm regards
> >
> > Yiming Liu (刘一鸣)
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
>
--
With Warm regards
Yiming Liu (刘一鸣)
Re: Turn off some hive configuration commands when they are not allowed
Posted by hongbin ma <ma...@apache.org>.
Do we have JIRAs to track the issue? it's easy to forget
On Sat, Sep 3, 2016 at 5:57 PM, Yiming Liu <li...@gmail.com> wrote:
> No progress, pending.
>
> 2016-09-03 17:36 GMT+08:00 hongbin ma <ma...@apache.org>:
>
> > do we have any progress on such documents?
> >
> > On Tue, Aug 2, 2016 at 8:50 AM, Yiming Liu <li...@gmail.com>
> > wrote:
> >
> > > Thanks, Shaofeng. It makes sense to grant enough privileges to Kylin
> for
> > > Cube building. Just in some extreme cases, the privilege issue will be
> a
> > > show stop.
> > >
> > > The privilege document is great. It's very helpful for Hadoop system
> > > administrator.
> > >
> > > 2016-08-01 9:38 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> > >
> > > > Hi Yiming,
> > > >
> > > > The "mapreduce.job.reduces" need by set at runtime, whose number is
> > > > calculated based on user tables' size, it couldn't be pre-configured.
> > > >
> > > > The "hive.merge.mapredfiles=false" can be externalized to the conf
> > file;
> > > > The hive merge is not needed since 1.5.3, I set in code to ensure it
> > will
> > > > be not be enabled (config files before 1.5.3 has this param set to
> > true).
> > > >
> > > > For other parameters, I think they're optional, but it is better to
> > keep
> > > as
> > > > they're good for performance, like dfs.replication=2, compress.codec
> > etc.
> > > >
> > > > Usually in a hadoop cluster, Apache Kylin should be treated as a
> > > > priviledged user (instead of a normal user like analyst), which can
> > > execute
> > > > necessary hadoop/hdfs/hbase/hive actions (like mkdir, create htable,
> > > etc);
> > > > To achieve this, the administartor need do some configurations and
> > > > authorizations; What we need do is to compose a document to list
> > > > these privileges, what's your opinion?
> > > >
> > > > Thanks for the comment!
> > > >
> > > >
> > > > 2016-07-30 14:03 GMT+08:00 Yiming Liu <li...@gmail.com>:
> > > >
> > > > > Hi Kylin dev,
> > > > >
> > > > > The first step is building cube is to CreateFlatHiveTable, it will
> > > call a
> > > > > few hive configuration commands, such as
> > > > > CreateFlatHiveTableStep line 78 and 79.
> > > > > set mapreduce.job.reduces=numReduces
> > > > > set hive.merge.mapredfiles=false
> > > > >
> > > > > Are these commands necessary for the cube building? Could we
> > configure
> > > > them
> > > > > in files? I met some cases, where the hiveserver would say
> > > "Configuration
> > > > > is not allowed to modify at runtime". It will break the build.
> > > > >
> > > > > Maybe there are some other hard code hadoop commands still. It will
> > be
> > > > more
> > > > > friendly if they could turn off on demand.
> > > > >
> > > > > --
> > > > > With Warm regards
> > > > >
> > > > > Yiming Liu (刘一鸣)
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > > Shaofeng Shi
> > > >
> > >
> > >
> > >
> > > --
> > > With Warm regards
> > >
> > > Yiming Liu (刘一鸣)
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> >
>
>
>
> --
> With Warm regards
>
> Yiming Liu (刘一鸣)
>
--
Regards,
*Bin Mahone | 马洪宾*
Re: Turn off some hive configuration commands when they are not allowed
Posted by Yiming Liu <li...@gmail.com>.
No progress, pending.
2016-09-03 17:36 GMT+08:00 hongbin ma <ma...@apache.org>:
> do we have any progress on such documents?
>
> On Tue, Aug 2, 2016 at 8:50 AM, Yiming Liu <li...@gmail.com>
> wrote:
>
> > Thanks, Shaofeng. It makes sense to grant enough privileges to Kylin for
> > Cube building. Just in some extreme cases, the privilege issue will be a
> > show stop.
> >
> > The privilege document is great. It's very helpful for Hadoop system
> > administrator.
> >
> > 2016-08-01 9:38 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> >
> > > Hi Yiming,
> > >
> > > The "mapreduce.job.reduces" need by set at runtime, whose number is
> > > calculated based on user tables' size, it couldn't be pre-configured.
> > >
> > > The "hive.merge.mapredfiles=false" can be externalized to the conf
> file;
> > > The hive merge is not needed since 1.5.3, I set in code to ensure it
> will
> > > be not be enabled (config files before 1.5.3 has this param set to
> true).
> > >
> > > For other parameters, I think they're optional, but it is better to
> keep
> > as
> > > they're good for performance, like dfs.replication=2, compress.codec
> etc.
> > >
> > > Usually in a hadoop cluster, Apache Kylin should be treated as a
> > > priviledged user (instead of a normal user like analyst), which can
> > execute
> > > necessary hadoop/hdfs/hbase/hive actions (like mkdir, create htable,
> > etc);
> > > To achieve this, the administartor need do some configurations and
> > > authorizations; What we need do is to compose a document to list
> > > these privileges, what's your opinion?
> > >
> > > Thanks for the comment!
> > >
> > >
> > > 2016-07-30 14:03 GMT+08:00 Yiming Liu <li...@gmail.com>:
> > >
> > > > Hi Kylin dev,
> > > >
> > > > The first step is building cube is to CreateFlatHiveTable, it will
> > call a
> > > > few hive configuration commands, such as
> > > > CreateFlatHiveTableStep line 78 and 79.
> > > > set mapreduce.job.reduces=numReduces
> > > > set hive.merge.mapredfiles=false
> > > >
> > > > Are these commands necessary for the cube building? Could we
> configure
> > > them
> > > > in files? I met some cases, where the hiveserver would say
> > "Configuration
> > > > is not allowed to modify at runtime". It will break the build.
> > > >
> > > > Maybe there are some other hard code hadoop commands still. It will
> be
> > > more
> > > > friendly if they could turn off on demand.
> > > >
> > > > --
> > > > With Warm regards
> > > >
> > > > Yiming Liu (刘一鸣)
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > Shaofeng Shi
> > >
> >
> >
> >
> > --
> > With Warm regards
> >
> > Yiming Liu (刘一鸣)
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
>
--
With Warm regards
Yiming Liu (刘一鸣)
Re: Turn off some hive configuration commands when they are not allowed
Posted by hongbin ma <ma...@apache.org>.
do we have any progress on such documents?
On Tue, Aug 2, 2016 at 8:50 AM, Yiming Liu <li...@gmail.com> wrote:
> Thanks, Shaofeng. It makes sense to grant enough privileges to Kylin for
> Cube building. Just in some extreme cases, the privilege issue will be a
> show stop.
>
> The privilege document is great. It's very helpful for Hadoop system
> administrator.
>
> 2016-08-01 9:38 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>
> > Hi Yiming,
> >
> > The "mapreduce.job.reduces" need by set at runtime, whose number is
> > calculated based on user tables' size, it couldn't be pre-configured.
> >
> > The "hive.merge.mapredfiles=false" can be externalized to the conf file;
> > The hive merge is not needed since 1.5.3, I set in code to ensure it will
> > be not be enabled (config files before 1.5.3 has this param set to true).
> >
> > For other parameters, I think they're optional, but it is better to keep
> as
> > they're good for performance, like dfs.replication=2, compress.codec etc.
> >
> > Usually in a hadoop cluster, Apache Kylin should be treated as a
> > priviledged user (instead of a normal user like analyst), which can
> execute
> > necessary hadoop/hdfs/hbase/hive actions (like mkdir, create htable,
> etc);
> > To achieve this, the administartor need do some configurations and
> > authorizations; What we need do is to compose a document to list
> > these privileges, what's your opinion?
> >
> > Thanks for the comment!
> >
> >
> > 2016-07-30 14:03 GMT+08:00 Yiming Liu <li...@gmail.com>:
> >
> > > Hi Kylin dev,
> > >
> > > The first step is building cube is to CreateFlatHiveTable, it will
> call a
> > > few hive configuration commands, such as
> > > CreateFlatHiveTableStep line 78 and 79.
> > > set mapreduce.job.reduces=numReduces
> > > set hive.merge.mapredfiles=false
> > >
> > > Are these commands necessary for the cube building? Could we configure
> > them
> > > in files? I met some cases, where the hiveserver would say
> "Configuration
> > > is not allowed to modify at runtime". It will break the build.
> > >
> > > Maybe there are some other hard code hadoop commands still. It will be
> > more
> > > friendly if they could turn off on demand.
> > >
> > > --
> > > With Warm regards
> > >
> > > Yiming Liu (刘一鸣)
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi
> >
>
>
>
> --
> With Warm regards
>
> Yiming Liu (刘一鸣)
>
--
Regards,
*Bin Mahone | 马洪宾*
Re: Turn off some hive configuration commands when they are not allowed
Posted by Yiming Liu <li...@gmail.com>.
Thanks, Shaofeng. It makes sense to grant enough privileges to Kylin for
Cube building. Just in some extreme cases, the privilege issue will be a
show stop.
The privilege document is great. It's very helpful for Hadoop system
administrator.
2016-08-01 9:38 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> Hi Yiming,
>
> The "mapreduce.job.reduces" need by set at runtime, whose number is
> calculated based on user tables' size, it couldn't be pre-configured.
>
> The "hive.merge.mapredfiles=false" can be externalized to the conf file;
> The hive merge is not needed since 1.5.3, I set in code to ensure it will
> be not be enabled (config files before 1.5.3 has this param set to true).
>
> For other parameters, I think they're optional, but it is better to keep as
> they're good for performance, like dfs.replication=2, compress.codec etc.
>
> Usually in a hadoop cluster, Apache Kylin should be treated as a
> priviledged user (instead of a normal user like analyst), which can execute
> necessary hadoop/hdfs/hbase/hive actions (like mkdir, create htable, etc);
> To achieve this, the administartor need do some configurations and
> authorizations; What we need do is to compose a document to list
> these privileges, what's your opinion?
>
> Thanks for the comment!
>
>
> 2016-07-30 14:03 GMT+08:00 Yiming Liu <li...@gmail.com>:
>
> > Hi Kylin dev,
> >
> > The first step is building cube is to CreateFlatHiveTable, it will call a
> > few hive configuration commands, such as
> > CreateFlatHiveTableStep line 78 and 79.
> > set mapreduce.job.reduces=numReduces
> > set hive.merge.mapredfiles=false
> >
> > Are these commands necessary for the cube building? Could we configure
> them
> > in files? I met some cases, where the hiveserver would say "Configuration
> > is not allowed to modify at runtime". It will break the build.
> >
> > Maybe there are some other hard code hadoop commands still. It will be
> more
> > friendly if they could turn off on demand.
> >
> > --
> > With Warm regards
> >
> > Yiming Liu (刘一鸣)
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
--
With Warm regards
Yiming Liu (刘一鸣)
Re: Turn off some hive configuration commands when they are not allowed
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Yiming,
The "mapreduce.job.reduces" need by set at runtime, whose number is
calculated based on user tables' size, it couldn't be pre-configured.
The "hive.merge.mapredfiles=false" can be externalized to the conf file;
The hive merge is not needed since 1.5.3, I set in code to ensure it will
be not be enabled (config files before 1.5.3 has this param set to true).
For other parameters, I think they're optional, but it is better to keep as
they're good for performance, like dfs.replication=2, compress.codec etc.
Usually in a hadoop cluster, Apache Kylin should be treated as a
priviledged user (instead of a normal user like analyst), which can execute
necessary hadoop/hdfs/hbase/hive actions (like mkdir, create htable, etc);
To achieve this, the administartor need do some configurations and
authorizations; What we need do is to compose a document to list
these privileges, what's your opinion?
Thanks for the comment!
2016-07-30 14:03 GMT+08:00 Yiming Liu <li...@gmail.com>:
> Hi Kylin dev,
>
> The first step is building cube is to CreateFlatHiveTable, it will call a
> few hive configuration commands, such as
> CreateFlatHiveTableStep line 78 and 79.
> set mapreduce.job.reduces=numReduces
> set hive.merge.mapredfiles=false
>
> Are these commands necessary for the cube building? Could we configure them
> in files? I met some cases, where the hiveserver would say "Configuration
> is not allowed to modify at runtime". It will break the build.
>
> Maybe there are some other hard code hadoop commands still. It will be more
> friendly if they could turn off on demand.
>
> --
> With Warm regards
>
> Yiming Liu (刘一鸣)
>
--
Best regards,
Shaofeng Shi