You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by ro...@stratebi.com on 2018/02/23 14:17:11 UTC
YARN Queues and Kylin
Hi,
There is any way to configure one cube to be built using a chosen YARN
queue?
The only way I see is to add the following property on
kylin_job_conf_inmem.xml and kylin_job_conf.xml, but this affects all cube
and not just one, as I need.
<property>
<name>mapreduce.job.queuename</name>
<value>Kylin</value>
</property>
</configuration>
Thanks in advance,
Roberto Tardío Olmos
Senior Big Data & Business Intelligence Consultant
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10
http://bigdata.stratebi.com/
http://www.stratebi.com
Re: YARN Queues and Kylin
Posted by Billy Liu <bi...@apache.org>.
Thanks Li Feng. Good practice.
With Warm regards
Billy Liu
2018-02-26 10:24 GMT+08:00 Li Feng <li...@kyligence.io>:
> HI Robeto,
>
>
>
> Sorry.I tried again and found the newly version can edit configuration
> overwrite before building.
>
> So you can change queue before building.
>
> For restful interface, you can operate in GUI and see the detail in Chrome
> console:
>
>
>
>
>
> BR,
>
> Lee.
>
>
>
> *发件人**: *Li Feng <li...@kyligence.io>
> *答复**: *"user@kylin.apache.org" <us...@kylin.apache.org>
> *日期**: *2018年2月26日 星期一 09:55
> *至**: *"user@kylin.apache.org" <us...@kylin.apache.org>
> *主**题**: *Re: YARN Queues and Kylin
>
>
>
> HI Robeto,
>
>
>
> I don’t think choosing the queue in building time is realizable in this
> version.Apache Kylin didn’t support change this setting during building.
>
> A work around may be manually using yarn command to change queue:
>
> yarn application –movetoqueue *application_Id *–queue* queue_name*
>
>
>
> BR,
>
> Lee.
>
>
>
> *发件人: *"roberto.tardio@stratebi.com" <ro...@stratebi.com>
> *答复: *"user@kylin.apache.org" <us...@kylin.apache.org>
> *日期: *2018年2月24日 星期六 18:31
> *至: *"user@kylin.apache.org" <us...@kylin.apache.org>
> *主**题: *RE: YARN Queues and Kylin
>
>
>
> Many thanks Li Feng!!
>
>
>
> This configuration is very useful for shared Hadoop enviroments.
>
>
>
> Do you think also could be useful to choose yarn queue at cube build time?
>
> E.g. If you have at same time an historical built process over this cube
> while and incremental process is also populating data in the same cube.
> Choosing the queue in building time (e.g. Rest api command) could help the
> incremental load not be delayed. What do you think about?
>
>
>
> Best regards,
>
>
>
> *From:* Li Feng [mailto:li.feng@kyligence.io]
> *Sent:* sábado, 24 de febrero de 2018 3:24
> *To:* user@kylin.apache.org
> *Subject:* Re: YARN Queues and Kylin
>
>
>
> HI Roberto,
>
> You can set the YARN Queue in Cube’s Configuration Overwrites page, then
> it will affect only one cube.
>
>
>
> kylin.engine.mr.config-override.mapreduce.job.queuename = YOUR_QUEUE_NAME
>
> kylin.source.hive.config-override.mapreduce.job.queuename =
> YOUR_QUEUE_NAME
>
>
>
>
>
> BR,
>
> Lee.
>
>
>
> *发件人: *"Kumar, Manoj H" <ma...@jpmorgan.com>
> *答复: *"user@kylin.apache.org" <us...@kylin.apache.org>
> *日期: *2018年2月23日 星期五 22:23
> *至: *"user@kylin.apache.org" <us...@kylin.apache.org>
> *主**题: *RE: YARN Queues and Kylin
>
>
>
> It will be for all cubes as these properties are defined at Kylin levels.
>
>
> Sent with BlackBerry Work (www.blackberry.com)
> ------------------------------
>
> *From: *roberto.tardio@stratebi.com
> *Sent: *23-Feb-2018 7:47 PM
> *To: *user@kylin.apache.org
> *Subject: *YARN Queues and Kylin
>
> Hi,
>
>
>
> There is any way to configure one cube to be built using a chosen YARN
> queue?
>
>
>
> The only way I see is to add the following property on
> kylin_job_conf_inmem.xml and kylin_job_conf.xml, but this affects all cube
> and not just one, as I need.
>
>
>
> <property>
>
> <name>mapreduce.job.queuename</name>
>
> <value>Kylin</value>
>
> </property>
>
> </configuration>
>
>
>
> Thanks in advance,
>
> *Roberto Tardío Olmos*
>
> *Senior Big Data & Business Intelligence Consultant*
>
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
>
> Fijo: 91.788.34.10
>
>
>
>
> http://bigdata.stratebi.com/
>
>
>
> http://www.stratebi.com
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
Re: YARN Queues and Kylin
Posted by Li Feng <li...@kyligence.io>.
HI Robeto,
Sorry.I tried again and found the newly version can edit configuration overwrite before building.
So you can change queue before building.
For restful interface, you can operate in GUI and see the detail in Chrome console:
[cid:image001.png@01D3AEEC.04D26AD0]
BR,
Lee.
发件人: Li Feng <li...@kyligence.io>
答复: "user@kylin.apache.org" <us...@kylin.apache.org>
日期: 2018年2月26日 星期一 09:55
至: "user@kylin.apache.org" <us...@kylin.apache.org>
主题: Re: YARN Queues and Kylin
HI Robeto,
I don’t think choosing the queue in building time is realizable in this version.Apache Kylin didn’t support change this setting during building.
A work around may be manually using yarn command to change queue:
yarn application –movetoqueue application_Id –queue queue_name
BR,
Lee.
发件人: "roberto.tardio@stratebi.com" <ro...@stratebi.com>
答复: "user@kylin.apache.org" <us...@kylin.apache.org>
日期: 2018年2月24日 星期六 18:31
至: "user@kylin.apache.org" <us...@kylin.apache.org>
主题: RE: YARN Queues and Kylin
Many thanks Li Feng!!
This configuration is very useful for shared Hadoop enviroments.
Do you think also could be useful to choose yarn queue at cube build time?
E.g. If you have at same time an historical built process over this cube while and incremental process is also populating data in the same cube. Choosing the queue in building time (e.g. Rest api command) could help the incremental load not be delayed. What do you think about?
Best regards,
From: Li Feng [mailto:li.feng@kyligence.io]
Sent: sábado, 24 de febrero de 2018 3:24
To: user@kylin.apache.org
Subject: Re: YARN Queues and Kylin
HI Roberto,
You can set the YARN Queue in Cube’s Configuration Overwrites page, then it will affect only one cube.
kylin.engine.mr.config-override.mapreduce.job.queuename = YOUR_QUEUE_NAME
kylin.source.hive.config-override.mapreduce.job.queuename = YOUR_QUEUE_NAME
[cid:image002.png@01D3AEEC.04D26AD0]
BR,
Lee.
发件人: "Kumar, Manoj H" <ma...@jpmorgan.com>>
答复: "user@kylin.apache.org<ma...@kylin.apache.org>" <us...@kylin.apache.org>>
日期: 2018年2月23日 星期五 22:23
至: "user@kylin.apache.org<ma...@kylin.apache.org>" <us...@kylin.apache.org>>
主题: RE: YARN Queues and Kylin
It will be for all cubes as these properties are defined at Kylin levels.
Sent with BlackBerry Work (www.blackberry.com<http://www.blackberry.com>)
________________________________
From: roberto.tardio@stratebi.com<ma...@stratebi.com>
Sent: 23-Feb-2018 7:47 PM
To: user@kylin.apache.org<ma...@kylin.apache.org>
Subject: YARN Queues and Kylin
Hi,
There is any way to configure one cube to be built using a chosen YARN queue?
The only way I see is to add the following property on kylin_job_conf_inmem.xml and kylin_job_conf.xml, but this affects all cube and not just one, as I need.
<property>
<name>mapreduce.job.queuename</name>
<value>Kylin</value>
</property>
</configuration>
Thanks in advance,
Roberto Tardío Olmos
Senior Big Data & Business Intelligence Consultant
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10
[cid:image003.png@01D3AEEC.04D26AD0]
http://bigdata.stratebi.com/
http://www.stratebi.com
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: YARN Queues and Kylin
Posted by Li Feng <li...@kyligence.io>.
HI Robeto,
I don’t think choosing the queue in building time is realizable in this version.Apache Kylin didn’t support change this setting during building.
A work around may be manually using yarn command to change queue:
yarn application –movetoqueue application_Id –queue queue_name
BR,
Lee.
发件人: "roberto.tardio@stratebi.com" <ro...@stratebi.com>
答复: "user@kylin.apache.org" <us...@kylin.apache.org>
日期: 2018年2月24日 星期六 18:31
至: "user@kylin.apache.org" <us...@kylin.apache.org>
主题: RE: YARN Queues and Kylin
Many thanks Li Feng!!
This configuration is very useful for shared Hadoop enviroments.
Do you think also could be useful to choose yarn queue at cube build time?
E.g. If you have at same time an historical built process over this cube while and incremental process is also populating data in the same cube. Choosing the queue in building time (e.g. Rest api command) could help the incremental load not be delayed. What do you think about?
Best regards,
From: Li Feng [mailto:li.feng@kyligence.io]
Sent: sábado, 24 de febrero de 2018 3:24
To: user@kylin.apache.org
Subject: Re: YARN Queues and Kylin
HI Roberto,
You can set the YARN Queue in Cube’s Configuration Overwrites page, then it will affect only one cube.
kylin.engine.mr.config-override.mapreduce.job.queuename = YOUR_QUEUE_NAME
kylin.source.hive.config-override.mapreduce.job.queuename = YOUR_QUEUE_NAME
[cid:image001.png@01D3AEE7.E2B1D520]
BR,
Lee.
发件人: "Kumar, Manoj H" <ma...@jpmorgan.com>>
答复: "user@kylin.apache.org<ma...@kylin.apache.org>" <us...@kylin.apache.org>>
日期: 2018年2月23日 星期五 22:23
至: "user@kylin.apache.org<ma...@kylin.apache.org>" <us...@kylin.apache.org>>
主题: RE: YARN Queues and Kylin
It will be for all cubes as these properties are defined at Kylin levels.
Sent with BlackBerry Work (www.blackberry.com<http://www.blackberry.com>)
________________________________
From: roberto.tardio@stratebi.com<ma...@stratebi.com>
Sent: 23-Feb-2018 7:47 PM
To: user@kylin.apache.org<ma...@kylin.apache.org>
Subject: YARN Queues and Kylin
Hi,
There is any way to configure one cube to be built using a chosen YARN queue?
The only way I see is to add the following property on kylin_job_conf_inmem.xml and kylin_job_conf.xml, but this affects all cube and not just one, as I need.
<property>
<name>mapreduce.job.queuename</name>
<value>Kylin</value>
</property>
</configuration>
Thanks in advance,
Roberto Tardío Olmos
Senior Big Data & Business Intelligence Consultant
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10
[tp://www.stratebi.com/image/layout_set_logo?img_id=21615&t=1486381163544]
http://bigdata.stratebi.com/
http://www.stratebi.com
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
RE: YARN Queues and Kylin
Posted by ro...@stratebi.com.
Many thanks Li Feng!!
This configuration is very useful for shared Hadoop enviroments.
Do you think also could be useful to choose yarn queue at cube build time?
E.g. If you have at same time an historical built process over this cube while and incremental process is also populating data in the same cube. Choosing the queue in building time (e.g. Rest api command) could help the incremental load not be delayed. What do you think about?
Best regards,
From: Li Feng [mailto:li.feng@kyligence.io]
Sent: sábado, 24 de febrero de 2018 3:24
To: user@kylin.apache.org
Subject: Re: YARN Queues and Kylin
HI Roberto,
You can set the YARN Queue in Cube’s Configuration Overwrites page, then it will affect only one cube.
kylin.engine.mr.config-override.mapreduce.job.queuename = YOUR_QUEUE_NAME
kylin.source.hive.config-override.mapreduce.job.queuename = YOUR_QUEUE_NAME
BR,
Lee.
发件人: "Kumar, Manoj H" <manoj.h.kumar@jpmorgan.com <ma...@jpmorgan.com> >
答复: "user@kylin.apache.org <ma...@kylin.apache.org> " <user@kylin.apache.org <ma...@kylin.apache.org> >
日期: 2018年2月23日 星期五 22:23
至: "user@kylin.apache.org <ma...@kylin.apache.org> " <user@kylin.apache.org <ma...@kylin.apache.org> >
主题: RE: YARN Queues and Kylin
It will be for all cubes as these properties are defined at Kylin levels.
Sent with BlackBerry Work (www.blackberry.com <http://www.blackberry.com> )
_____
From: roberto.tardio@stratebi.com <ma...@stratebi.com>
Sent: 23-Feb-2018 7:47 PM
To: user@kylin.apache.org <ma...@kylin.apache.org>
Subject: YARN Queues and Kylin
Hi,
There is any way to configure one cube to be built using a chosen YARN queue?
The only way I see is to add the following property on kylin_job_conf_inmem.xml and kylin_job_conf.xml, but this affects all cube and not just one, as I need.
<property>
<name>mapreduce.job.queuename</name>
<value>Kylin</value>
</property>
</configuration>
Thanks in advance,
Roberto Tardío Olmos
Senior Big Data & Business Intelligence Consultant
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10
http://bigdata.stratebi.com/
http://www.stratebi.com
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer <http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: YARN Queues and Kylin
Posted by Li Feng <li...@kyligence.io>.
HI Roberto,
You can set the YARN Queue in Cube’s Configuration Overwrites page, then it will affect only one cube.
kylin.engine.mr.config-override.mapreduce.job.queuename = YOUR_QUEUE_NAME
kylin.source.hive.config-override.mapreduce.job.queuename = YOUR_QUEUE_NAME
[cid:image001.png@01D3AD59.8958D030]
BR,
Lee.
发件人: "Kumar, Manoj H" <ma...@jpmorgan.com>
答复: "user@kylin.apache.org" <us...@kylin.apache.org>
日期: 2018年2月23日 星期五 22:23
至: "user@kylin.apache.org" <us...@kylin.apache.org>
主题: RE: YARN Queues and Kylin
It will be for all cubes as these properties are defined at Kylin levels.
Sent with BlackBerry Work (www.blackberry.com)
________________________________
From: roberto.tardio@stratebi.com<ma...@stratebi.com>
Sent: 23-Feb-2018 7:47 PM
To: user@kylin.apache.org<ma...@kylin.apache.org>
Subject: YARN Queues and Kylin
Hi,
There is any way to configure one cube to be built using a chosen YARN queue?
The only way I see is to add the following property on kylin_job_conf_inmem.xml and kylin_job_conf.xml, but this affects all cube and not just one, as I need.
<property>
<name>mapreduce.job.queuename</name>
<value>Kylin</value>
</property>
</configuration>
Thanks in advance,
Roberto Tardío Olmos
Senior Big Data & Business Intelligence Consultant
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10
[ttp://www.stratebi.com/image/layout_set_logo?img_id=21615&t=1486381163544]
http://bigdata.stratebi.com/
http://www.stratebi.com
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
RE: YARN Queues and Kylin
Posted by "Kumar, Manoj H" <ma...@jpmorgan.com>.
It will be for all cubes as these properties are defined at Kylin levels.
Sent with BlackBerry Work (www.blackberry.com)
________________________________
From: roberto.tardio@stratebi.com
Sent: 23-Feb-2018 7:47 PM
To: user@kylin.apache.org
Subject: YARN Queues and Kylin
Hi,
There is any way to configure one cube to be built using a chosen YARN queue?
The only way I see is to add the following property on kylin_job_conf_inmem.xml and kylin_job_conf.xml, but this affects all cube and not just one, as I need.
<property>
<name>mapreduce.job.queuename</name>
<value>Kylin</value>
</property>
</configuration>
Thanks in advance,
Roberto Tardío Olmos
Senior Big Data & Business Intelligence Consultant
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10
[http://www.stratebi.com/image/layout_set_logo?img_id=21615&t=1486381163544]
http://bigdata.stratebi.com/<https://secureweb.jpmchase.net/readonly/http://bigdata.stratebi.com/>
http://www.stratebi.com<https://secureweb.jpmchase.net/readonly/http://www.stratebi.com>
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.