You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Alberto Ramón <a....@gmail.com> on 2016/10/29 17:42:20 UTC
Kylin Dependencies
Hi
Target:
All Kylin docker are VERY heavy !! (GB and hundred of process) ---> That
Is Good for Develop / testing , but BAD Idea for production
I'm trying to install Kylin on minimal linux, ideally Alpine or similar
I have:
- a clean install of linux (minimal Centos for example) , without Hadoop,
and and install Kylin from binary
- use remote HBase & Hive
Which dependencies of Kylin I Will need on my Centos / Alpine?
BR, Alb
Re: Kylin Dependencies
Posted by ShaoFeng Shi <sh...@apache.org>.
Alberto, thanks for the info, that is valuable for making a new image;
looking forward for the update!
2016-11-03 16:25 GMT+08:00 Alberto Ramón <a....@gmail.com>:
> yes, I saw 1.200 process
> (this can be OK for All in One Docker , for developing or training
> process)
>
> I showed this to a "Docker Captain" .... and (was funny)
>
> (
> Docker is some radical , also:
> Understand that there is only 1 user process by container , with PID = 1
> And only monitorice/check PID = 1
> If PID 1 is ok --> container is OK
> if PID 1 is no OK, --> auto restart container
>
> Which process will be PID 1? The process that you put CMD in dockerfile
> (obviously only can have one CMD per dockerfile)
> )
>
> I will try to build a "MIni Kylin", ... I will info you about my progress
> ¡¡
>
> 2016-11-03 6:49 GMT+01:00 ShaoFeng Shi <sh...@apache.org>:
>
>> are there so many running processes in the kylin docker image? Although
>> Kylin relies on Hive/YARN/HDFS/HBase, they are all client jars instead of
>> running services; To minimal the docker image, some components can be
>> removed like openssh-server openssh-clients snappy snappy-devel
>> hadoop-native (from 1.5.4 kylin doesn't use compression by default)
>>
>> 2016-11-02 23:15 GMT+08:00 Alberto Ramón <a....@gmail.com>:
>>
>>> yes, I tested (and use) this and other previous version
>>>
>>> BUT the image :
>>> - more than > 1000 process
>>> - more than > 3GB
>>>
>>> This is OK (very OK) for testing / develop / PoC
>>>
>>> But for production (docker recomendations):
>>> - Ideally 1 process (5-10 can be acceptable)
>>> - < 100 MB (200 - 300 MB can be acceptable)
>>>
>>> The target is: Create Kylin docker (minimal) *with out install *Hive,
>>> YARN, HDFS, or HBase ....
>>>
>>> 2016-11-02 15:52 GMT+01:00 Billy(Yiming) Liu <li...@gmail.com>:
>>>
>>>> Here is a quick start for running Kylin on docker,
>>>> https://github.com/kyligence/kylin-docker
>>>>
>>>> From the docker file, you could find the kylin dependencies.
>>>>
>>>> 2016-11-02 22:46 GMT+08:00 Alberto Ramón <a....@gmail.com>:
>>>>
>>>>> With configs ....... I can try it (Will be an interesting exercise
>>>>> for me)
>>>>> But libraries, ...
>>>>> These libraries can be static compiled on Kylin?
>>>>> Any Idea / solution about how to solve all dependecies with out
>>>>> install HDFS, Yarn, Hive, HBase in this minimal Linux... ?
>>>>>
>>>>> the idea is make "minimal linux + Kylin" "to docker it"
>>>>> (The result must be few MB, < 150 MB)
>>>>>
>>>>>
>>>>> 2016-11-02 14:20 GMT+01:00 Li Yang <li...@apache.org>:
>>>>>
>>>>>> Kylin needs Hadoop client library and configs, including hdfs, yarn,
>>>>>> hive, hbase.
>>>>>>
>>>>>> On Sun, Oct 30, 2016 at 1:42 AM, Alberto Ramón <
>>>>>> a.ramonportoles@gmail.com> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> Target:
>>>>>>> All Kylin docker are VERY heavy !! (GB and hundred of process)
>>>>>>> ---> That Is Good for Develop / testing , but BAD Idea for production
>>>>>>> I'm trying to install Kylin on minimal linux, ideally Alpine or
>>>>>>> similar
>>>>>>>
>>>>>>> I have:
>>>>>>> - a clean install of linux (minimal Centos for example) , without
>>>>>>> Hadoop, and and install Kylin from binary
>>>>>>> - use remote HBase & Hive
>>>>>>>
>>>>>>>
>>>>>>> Which dependencies of Kylin I Will need on my Centos / Alpine?
>>>>>>>
>>>>>>> BR, Alb
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> With Warm regards
>>>>
>>>> Yiming Liu (刘一鸣)
>>>>
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Kylin Dependencies
Posted by Alberto Ramón <a....@gmail.com>.
yes, I saw 1.200 process
(this can be OK for All in One Docker , for developing or training process)
I showed this to a "Docker Captain" .... and (was funny)
(
Docker is some radical , also:
Understand that there is only 1 user process by container , with PID = 1
And only monitorice/check PID = 1
If PID 1 is ok --> container is OK
if PID 1 is no OK, --> auto restart container
Which process will be PID 1? The process that you put CMD in dockerfile
(obviously only can have one CMD per dockerfile)
)
I will try to build a "MIni Kylin", ... I will info you about my progress ¡¡
2016-11-03 6:49 GMT+01:00 ShaoFeng Shi <sh...@apache.org>:
> are there so many running processes in the kylin docker image? Although
> Kylin relies on Hive/YARN/HDFS/HBase, they are all client jars instead of
> running services; To minimal the docker image, some components can be
> removed like openssh-server openssh-clients snappy snappy-devel
> hadoop-native (from 1.5.4 kylin doesn't use compression by default)
>
> 2016-11-02 23:15 GMT+08:00 Alberto Ramón <a....@gmail.com>:
>
>> yes, I tested (and use) this and other previous version
>>
>> BUT the image :
>> - more than > 1000 process
>> - more than > 3GB
>>
>> This is OK (very OK) for testing / develop / PoC
>>
>> But for production (docker recomendations):
>> - Ideally 1 process (5-10 can be acceptable)
>> - < 100 MB (200 - 300 MB can be acceptable)
>>
>> The target is: Create Kylin docker (minimal) *with out install *Hive,
>> YARN, HDFS, or HBase ....
>>
>> 2016-11-02 15:52 GMT+01:00 Billy(Yiming) Liu <li...@gmail.com>:
>>
>>> Here is a quick start for running Kylin on docker,
>>> https://github.com/kyligence/kylin-docker
>>>
>>> From the docker file, you could find the kylin dependencies.
>>>
>>> 2016-11-02 22:46 GMT+08:00 Alberto Ramón <a....@gmail.com>:
>>>
>>>> With configs ....... I can try it (Will be an interesting exercise for
>>>> me)
>>>> But libraries, ...
>>>> These libraries can be static compiled on Kylin?
>>>> Any Idea / solution about how to solve all dependecies with out
>>>> install HDFS, Yarn, Hive, HBase in this minimal Linux... ?
>>>>
>>>> the idea is make "minimal linux + Kylin" "to docker it"
>>>> (The result must be few MB, < 150 MB)
>>>>
>>>>
>>>> 2016-11-02 14:20 GMT+01:00 Li Yang <li...@apache.org>:
>>>>
>>>>> Kylin needs Hadoop client library and configs, including hdfs, yarn,
>>>>> hive, hbase.
>>>>>
>>>>> On Sun, Oct 30, 2016 at 1:42 AM, Alberto Ramón <
>>>>> a.ramonportoles@gmail.com> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Target:
>>>>>> All Kylin docker are VERY heavy !! (GB and hundred of process) --->
>>>>>> That Is Good for Develop / testing , but BAD Idea for production
>>>>>> I'm trying to install Kylin on minimal linux, ideally Alpine or
>>>>>> similar
>>>>>>
>>>>>> I have:
>>>>>> - a clean install of linux (minimal Centos for example) , without
>>>>>> Hadoop, and and install Kylin from binary
>>>>>> - use remote HBase & Hive
>>>>>>
>>>>>>
>>>>>> Which dependencies of Kylin I Will need on my Centos / Alpine?
>>>>>>
>>>>>> BR, Alb
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> With Warm regards
>>>
>>> Yiming Liu (刘一鸣)
>>>
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
Re: Kylin Dependencies
Posted by ShaoFeng Shi <sh...@apache.org>.
are there so many running processes in the kylin docker image? Although
Kylin relies on Hive/YARN/HDFS/HBase, they are all client jars instead of
running services; To minimal the docker image, some components can be
removed like openssh-server openssh-clients snappy snappy-devel
hadoop-native (from 1.5.4 kylin doesn't use compression by default)
2016-11-02 23:15 GMT+08:00 Alberto Ramón <a....@gmail.com>:
> yes, I tested (and use) this and other previous version
>
> BUT the image :
> - more than > 1000 process
> - more than > 3GB
>
> This is OK (very OK) for testing / develop / PoC
>
> But for production (docker recomendations):
> - Ideally 1 process (5-10 can be acceptable)
> - < 100 MB (200 - 300 MB can be acceptable)
>
> The target is: Create Kylin docker (minimal) *with out install *Hive,
> YARN, HDFS, or HBase ....
>
> 2016-11-02 15:52 GMT+01:00 Billy(Yiming) Liu <li...@gmail.com>:
>
>> Here is a quick start for running Kylin on docker, https://github.com/kyl
>> igence/kylin-docker
>>
>> From the docker file, you could find the kylin dependencies.
>>
>> 2016-11-02 22:46 GMT+08:00 Alberto Ramón <a....@gmail.com>:
>>
>>> With configs ....... I can try it (Will be an interesting exercise for
>>> me)
>>> But libraries, ...
>>> These libraries can be static compiled on Kylin?
>>> Any Idea / solution about how to solve all dependecies with out
>>> install HDFS, Yarn, Hive, HBase in this minimal Linux... ?
>>>
>>> the idea is make "minimal linux + Kylin" "to docker it"
>>> (The result must be few MB, < 150 MB)
>>>
>>>
>>> 2016-11-02 14:20 GMT+01:00 Li Yang <li...@apache.org>:
>>>
>>>> Kylin needs Hadoop client library and configs, including hdfs, yarn,
>>>> hive, hbase.
>>>>
>>>> On Sun, Oct 30, 2016 at 1:42 AM, Alberto Ramón <
>>>> a.ramonportoles@gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> Target:
>>>>> All Kylin docker are VERY heavy !! (GB and hundred of process) --->
>>>>> That Is Good for Develop / testing , but BAD Idea for production
>>>>> I'm trying to install Kylin on minimal linux, ideally Alpine or similar
>>>>>
>>>>> I have:
>>>>> - a clean install of linux (minimal Centos for example) , without
>>>>> Hadoop, and and install Kylin from binary
>>>>> - use remote HBase & Hive
>>>>>
>>>>>
>>>>> Which dependencies of Kylin I Will need on my Centos / Alpine?
>>>>>
>>>>> BR, Alb
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> With Warm regards
>>
>> Yiming Liu (刘一鸣)
>>
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Kylin Dependencies
Posted by Alberto Ramón <a....@gmail.com>.
yes, I tested (and use) this and other previous version
BUT the image :
- more than > 1000 process
- more than > 3GB
This is OK (very OK) for testing / develop / PoC
But for production (docker recomendations):
- Ideally 1 process (5-10 can be acceptable)
- < 100 MB (200 - 300 MB can be acceptable)
The target is: Create Kylin docker (minimal) *with out install *Hive,
YARN, HDFS, or HBase ....
2016-11-02 15:52 GMT+01:00 Billy(Yiming) Liu <li...@gmail.com>:
> Here is a quick start for running Kylin on docker, https://github.com/
> kyligence/kylin-docker
>
> From the docker file, you could find the kylin dependencies.
>
> 2016-11-02 22:46 GMT+08:00 Alberto Ramón <a....@gmail.com>:
>
>> With configs ....... I can try it (Will be an interesting exercise for
>> me)
>> But libraries, ...
>> These libraries can be static compiled on Kylin?
>> Any Idea / solution about how to solve all dependecies with out
>> install HDFS, Yarn, Hive, HBase in this minimal Linux... ?
>>
>> the idea is make "minimal linux + Kylin" "to docker it"
>> (The result must be few MB, < 150 MB)
>>
>>
>> 2016-11-02 14:20 GMT+01:00 Li Yang <li...@apache.org>:
>>
>>> Kylin needs Hadoop client library and configs, including hdfs, yarn,
>>> hive, hbase.
>>>
>>> On Sun, Oct 30, 2016 at 1:42 AM, Alberto Ramón <
>>> a.ramonportoles@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> Target:
>>>> All Kylin docker are VERY heavy !! (GB and hundred of process) --->
>>>> That Is Good for Develop / testing , but BAD Idea for production
>>>> I'm trying to install Kylin on minimal linux, ideally Alpine or similar
>>>>
>>>> I have:
>>>> - a clean install of linux (minimal Centos for example) , without
>>>> Hadoop, and and install Kylin from binary
>>>> - use remote HBase & Hive
>>>>
>>>>
>>>> Which dependencies of Kylin I Will need on my Centos / Alpine?
>>>>
>>>> BR, Alb
>>>>
>>>
>>>
>>
>
>
> --
> With Warm regards
>
> Yiming Liu (刘一鸣)
>
Re: Kylin Dependencies
Posted by "Billy(Yiming) Liu" <li...@gmail.com>.
Here is a quick start for running Kylin on docker,
https://github.com/kyligence/kylin-docker
From the docker file, you could find the kylin dependencies.
2016-11-02 22:46 GMT+08:00 Alberto Ramón <a....@gmail.com>:
> With configs ....... I can try it (Will be an interesting exercise for me)
> But libraries, ...
> These libraries can be static compiled on Kylin?
> Any Idea / solution about how to solve all dependecies with out
> install HDFS, Yarn, Hive, HBase in this minimal Linux... ?
>
> the idea is make "minimal linux + Kylin" "to docker it"
> (The result must be few MB, < 150 MB)
>
>
> 2016-11-02 14:20 GMT+01:00 Li Yang <li...@apache.org>:
>
>> Kylin needs Hadoop client library and configs, including hdfs, yarn,
>> hive, hbase.
>>
>> On Sun, Oct 30, 2016 at 1:42 AM, Alberto Ramón <a.ramonportoles@gmail.com
>> > wrote:
>>
>>> Hi
>>>
>>> Target:
>>> All Kylin docker are VERY heavy !! (GB and hundred of process) --->
>>> That Is Good for Develop / testing , but BAD Idea for production
>>> I'm trying to install Kylin on minimal linux, ideally Alpine or similar
>>>
>>> I have:
>>> - a clean install of linux (minimal Centos for example) , without
>>> Hadoop, and and install Kylin from binary
>>> - use remote HBase & Hive
>>>
>>>
>>> Which dependencies of Kylin I Will need on my Centos / Alpine?
>>>
>>> BR, Alb
>>>
>>
>>
>
--
With Warm regards
Yiming Liu (刘一鸣)
Re: Kylin Dependencies
Posted by Alberto Ramón <a....@gmail.com>.
With configs ....... I can try it (Will be an interesting exercise for me)
But libraries, ...
These libraries can be static compiled on Kylin?
Any Idea / solution about how to solve all dependecies with out install
HDFS, Yarn, Hive, HBase in this minimal Linux... ?
the idea is make "minimal linux + Kylin" "to docker it"
(The result must be few MB, < 150 MB)
2016-11-02 14:20 GMT+01:00 Li Yang <li...@apache.org>:
> Kylin needs Hadoop client library and configs, including hdfs, yarn, hive,
> hbase.
>
> On Sun, Oct 30, 2016 at 1:42 AM, Alberto Ramón <a....@gmail.com>
> wrote:
>
>> Hi
>>
>> Target:
>> All Kylin docker are VERY heavy !! (GB and hundred of process) --->
>> That Is Good for Develop / testing , but BAD Idea for production
>> I'm trying to install Kylin on minimal linux, ideally Alpine or similar
>>
>> I have:
>> - a clean install of linux (minimal Centos for example) , without Hadoop,
>> and and install Kylin from binary
>> - use remote HBase & Hive
>>
>>
>> Which dependencies of Kylin I Will need on my Centos / Alpine?
>>
>> BR, Alb
>>
>
>
Re: Kylin Dependencies
Posted by Li Yang <li...@apache.org>.
Kylin needs Hadoop client library and configs, including hdfs, yarn, hive,
hbase.
On Sun, Oct 30, 2016 at 1:42 AM, Alberto Ramón <a....@gmail.com>
wrote:
> Hi
>
> Target:
> All Kylin docker are VERY heavy !! (GB and hundred of process) ---> That
> Is Good for Develop / testing , but BAD Idea for production
> I'm trying to install Kylin on minimal linux, ideally Alpine or similar
>
> I have:
> - a clean install of linux (minimal Centos for example) , without Hadoop,
> and and install Kylin from binary
> - use remote HBase & Hive
>
>
> Which dependencies of Kylin I Will need on my Centos / Alpine?
>
> BR, Alb
>