You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by jxs <jx...@126.com> on 2017/11/15 02:21:37 UTC

Question about using dedicated EC2 server with EMR cluster

When using dedicated client node on EC2, should I configure kylin with the same:
kylin.env.hdfs-working-dir = s3://our-emr/kylin/
kylin.storage.hbase.cluster-fs = s3://our-emr


Should the kylin.server.mode be "query" ?


Thanks.

Re: Re: Re: Question about using dedicated EC2 server with EMR cluster

Posted by ShaoFeng Shi <sh...@apache.org>.
You're welcome!

2017-11-16 11:02 GMT+08:00 jxs <jx...@126.com>:

> Get it, quite clear, thank you.
>
>
>
> 在2017年11月16 10时52分, "ShaoFeng Shi"<sh...@apache.org>写道:
>
>
> "query" node runs the query engine, which can execute the SQL queries;
> "job" node will execute the cube build jobs, like run hive command, submit
> MR, update cube status, etc.
> "all" node will run both query engine and job engine.
>
> You can submit a job on a "query" node, but the job's execution only
> happens on the "job" or "all" node; they share the same meta store, "query"
> node will write the job task to meta store, "job" node will read it and
> then execute.
>
> Hope this clarifies.
>
> 2017-11-16 10:29 GMT+08:00 jxs <jx...@126.com>:
>
>> Hi, ShaoFeng, what the configuration "kylin.server.mode=query" means?
>> I saw the behaviour too yesterday that the "query" node submitted cube
>> and job operations.
>>
>>
>>
>> 在2017年11月16 09时13分, "ShaoFeng Shi"<sh...@apache.org>写道:
>>
>>
>> Even in a Hadoop client node (or "edge node"), Kylin can run the job
>> engine; The behavior of submitting a job to Hadoop can happen on any node;
>> you only need properly install/configure the Hadoop client and
>> configurations on it.
>>
>>
>> 2017-11-15 17:27 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
>>
>>> Hi Jxs,
>>>
>>>
>>> In our case we have not not detectted that kylin.server.mode
>>>  affects correct operating of Kylin in dedicated EC2 using an EMR
>>>  cluster.
>>>
>>> Kind regards,
>>>
>>>
>>>
>>> El 15/11/2017 a las 3:21, jxs escribió:
>>>
>>>
>>> When using dedicated client node on EC2, should I configure
>>>  kylin with the same:
>>> kylin.env.hdfs-working-dir = s3://our-emr/kylin/
>>>          kylin.storage.hbase.cluster-fs = s3://our-emr
>>>
>>>
>>> Should the kylin.server.mode be "query" ?
>>>
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> *Roberto Tardío Olmos*
>>> *Senior Big Data & Business          Intelligence Consultant*
>>> Avenida de Brasil, 17
>>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>>> Planta 16.28020        Madrid
>>>
>>> Fijo: 91.788.34.10
>>>
>>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>


-- 
Best regards,

Shaofeng Shi 史少锋

Re:Re: Re: Question about using dedicated EC2 server with EMR cluster

Posted by jxs <jx...@126.com>.
Get it, quite clear, thank you.







在2017年11月16 10时52分, "ShaoFeng Shi"<sh...@apache.org>写道:


"query" node runs the query engine, which can execute the SQL queries;
"job" node will execute the cube build jobs, like run hive command, submit MR, update cube status, etc.
"all" node will run both query engine and job engine.


You can submit a job on a "query" node, but the job's execution only happens on the "job" or "all" node; they share the same meta store, "query" node will write the job task to meta store, "job" node will read it and then execute.


Hope this clarifies.


2017-11-16 10:29 GMT+08:00 jxs <jx...@126.com>:

Hi, ShaoFeng, what the configuration "kylin.server.mode=query" means?
I saw the behaviour too yesterday that the "query" node submitted cube and job operations.







在2017年11月16 09时13分, "ShaoFeng Shi"<sh...@apache.org>写道:


Even in a Hadoop client node (or "edge node"), Kylin can run the job engine; The behavior of submitting a job to Hadoop can happen on any node; you only need properly install/configure the Hadoop client and configurations on it.




2017-11-15 17:27 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:


Hi Jxs,
   

In our case we have not not detectted that kylin.server.mode      affects correct operating of Kylin in dedicated EC2 using an EMR      cluster.

Kind regards,
   


   
El 15/11/2017 a las 3:21, jxs escribió:
   
When using dedicated client node on EC2, should I configure          kylin with the same:
kylin.env.hdfs-working-dir = s3://our-emr/kylin/
         kylin.storage.hbase.cluster-fs = s3://our-emr

       
Should the kylin.server.mode be "query" ?

       
Thanks.

       

       

       

   
--
     

Roberto Tardío Olmos

Senior Big Data & Business          Intelligence Consultant
Avenida de Brasil, 17, Planta 16.28020        Madrid
     
Fijo: 91.788.34.10
       





--

Best regards,


Shaofeng Shi 史少锋







--

Best regards,


Shaofeng Shi 史少锋

Re: Re: Question about using dedicated EC2 server with EMR cluster

Posted by ShaoFeng Shi <sh...@apache.org>.
"query" node runs the query engine, which can execute the SQL queries;
"job" node will execute the cube build jobs, like run hive command, submit
MR, update cube status, etc.
"all" node will run both query engine and job engine.

You can submit a job on a "query" node, but the job's execution only
happens on the "job" or "all" node; they share the same meta store, "query"
node will write the job task to meta store, "job" node will read it and
then execute.

Hope this clarifies.

2017-11-16 10:29 GMT+08:00 jxs <jx...@126.com>:

> Hi, ShaoFeng, what the configuration "kylin.server.mode=query" means?
> I saw the behaviour too yesterday that the "query" node submitted cube and
> job operations.
>
>
>
> 在2017年11月16 09时13分, "ShaoFeng Shi"<sh...@apache.org>写道:
>
>
> Even in a Hadoop client node (or "edge node"), Kylin can run the job
> engine; The behavior of submitting a job to Hadoop can happen on any node;
> you only need properly install/configure the Hadoop client and
> configurations on it.
>
>
> 2017-11-15 17:27 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
>
>> Hi Jxs,
>>
>>
>> In our case we have not not detectted that kylin.server.mode      affects
>> correct operating of Kylin in dedicated EC2 using an EMR      cluster.
>>
>> Kind regards,
>>
>>
>>
>> El 15/11/2017 a las 3:21, jxs escribió:
>>
>>
>> When using dedicated client node on EC2, should I configure
>>  kylin with the same:
>> kylin.env.hdfs-working-dir = s3://our-emr/kylin/
>>          kylin.storage.hbase.cluster-fs = s3://our-emr
>>
>>
>> Should the kylin.server.mode be "query" ?
>>
>>
>> Thanks.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>>
>> *Roberto Tardío Olmos*
>> *Senior Big Data & Business          Intelligence Consultant*
>> Avenida de Brasil, 17
>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>> Planta 16.28020        Madrid
>>
>> Fijo: 91.788.34.10
>>
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>


-- 
Best regards,

Shaofeng Shi 史少锋

Re:Re: Question about using dedicated EC2 server with EMR cluster

Posted by jxs <jx...@126.com>.
Hi, ShaoFeng, what the configuration "kylin.server.mode=query" means?
I saw the behaviour too yesterday that the "query" node submitted cube and job operations.







在2017年11月16 09时13分, "ShaoFeng Shi"<sh...@apache.org>写道:


Even in a Hadoop client node (or "edge node"), Kylin can run the job engine; The behavior of submitting a job to Hadoop can happen on any node; you only need properly install/configure the Hadoop client and configurations on it.




2017-11-15 17:27 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:


Hi Jxs,
   

In our case we have not not detectted that kylin.server.mode      affects correct operating of Kylin in dedicated EC2 using an EMR      cluster.

Kind regards,
   


   
El 15/11/2017 a las 3:21, jxs escribió:
   
When using dedicated client node on EC2, should I configure          kylin with the same:
kylin.env.hdfs-working-dir = s3://our-emr/kylin/
         kylin.storage.hbase.cluster-fs = s3://our-emr

       
Should the kylin.server.mode be "query" ?

       
Thanks.

       

       

       

   
--
     

Roberto Tardío Olmos

Senior Big Data & Business          Intelligence Consultant
Avenida de Brasil, 17, Planta 16.28020        Madrid
     
Fijo: 91.788.34.10
       





--

Best regards,


Shaofeng Shi 史少锋

Re: Question about using dedicated EC2 server with EMR cluster

Posted by ShaoFeng Shi <sh...@apache.org>.
Even in a Hadoop client node (or "edge node"), Kylin can run the job
engine; The behavior of submitting a job to Hadoop can happen on any node;
you only need properly install/configure the Hadoop client and
configurations on it.


2017-11-15 17:27 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:

> Hi Jxs,
>
> In our case we have not not detectted that kylin.server.mode affects
> correct operating of Kylin in dedicated EC2 using an EMR cluster.
>
> Kind regards,
>
> El 15/11/2017 a las 3:21, jxs escribió:
>
> When using dedicated client node on EC2, should I configure kylin with the
> same:
> kylin.env.hdfs-working-dir = s3://our-emr/kylin/
> kylin.storage.hbase.cluster-fs = s3://our-emr
>
> Should the kylin.server.mode be "query" ?
>
> Thanks.
>
>
>
>
> --
>
> *Roberto Tardío Olmos*
> *Senior Big Data & Business Intelligence Consultant*
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
> Fijo: 91.788.34.10
>



-- 
Best regards,

Shaofeng Shi 史少锋

Re: Question about using dedicated EC2 server with EMR cluster

Posted by Roberto Tardío <ro...@stratebi.com>.
Hi Jxs,

In our case we have not not detectted that kylin.server.mode affects 
correct operating of Kylin in dedicated EC2 using an EMR cluster.

Kind regards,


El 15/11/2017 a las 3:21, jxs escribió:
> When using dedicated client node on EC2, should I configure kylin with 
> the same:
> kylin.env.hdfs-working-dir = s3://our-emr/kylin/
> kylin.storage.hbase.cluster-fs = s3://our-emr
>
> Should the kylin.server.mode be "query" ?
>
> Thanks.
>
>
>

-- 

*Roberto Tardío Olmos*

/Senior Big Data & Business Intelligence Consultant/
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10