You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Harun Reşit Zafer <ha...@tubitak.gov.tr> on 2015/01/26 10:18:21 UTC
Multiple separate Hadoop clusters on same physical machines
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends
(Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop
(maybe different versions) and ecosystem on physical machines
(virtualization is not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every
team member play with his/her own cluster. However we have limited
amount of servers (strong machines though).
So the question is, by changing port numbers, environment variables and
other configuration parameters, is it possible to setup several
independent clusters on same physical machines. Is there any
constraints? What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268
W http://www.hrzafer.com
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
I see no possibility how federation may help to have different Clusters on _same_ machines. On top, federation isn’t production ready, since the NN can have massively issues with GC on high loaded systems, which will be the case here.
To have multiple, maybe single node, clusters the best way is to use cloud based solutions, e.g.. OpenStack with Docker containers. Also an mesos driven solution can help here, there are some good tutorials available.
BG,
Alexander
> On 26 Jan 2015, at 10:34, Azuryy Yu <az...@gmail.com> wrote:
>
> Hi,
>
> I think the best way is deploy HDFS federation with Hadoop 2.x.
>
> On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr>> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
> W http://www.hrzafer.com <http://www.hrzafer.com/>
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
I see no possibility how federation may help to have different Clusters on _same_ machines. On top, federation isn’t production ready, since the NN can have massively issues with GC on high loaded systems, which will be the case here.
To have multiple, maybe single node, clusters the best way is to use cloud based solutions, e.g.. OpenStack with Docker containers. Also an mesos driven solution can help here, there are some good tutorials available.
BG,
Alexander
> On 26 Jan 2015, at 10:34, Azuryy Yu <az...@gmail.com> wrote:
>
> Hi,
>
> I think the best way is deploy HDFS federation with Hadoop 2.x.
>
> On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr>> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
> W http://www.hrzafer.com <http://www.hrzafer.com/>
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by daemeon reiydelle <da...@gmail.com>.
Make virtualization an option. Federation will NOT solve your problems.
*.......*
*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Mon, Jan 26, 2015 at 1:34 AM, Azuryy Yu <az...@gmail.com> wrote:
> Hi,
>
> I think the best way is deploy HDFS federation with Hadoop 2.x.
>
> On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer <
> harun.zafer@tubitak.gov.tr> wrote:
>
>> Hi everyone,
>>
>> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
>> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
>> different versions) and ecosystem on physical machines (virtualization is
>> not an option) from different perspectives.
>>
>> As a bunch of developer we would like to work in parallel. We want every
>> team member play with his/her own cluster. However we have limited amount
>> of servers (strong machines though).
>>
>> So the question is, by changing port numbers, environment variables and
>> other configuration parameters, is it possible to setup several independent
>> clusters on same physical machines. Is there any constraints? What are the
>> possible difficulties we are to face?
>>
>> Thanks in advance
>>
>> --
>> Harun Reşit Zafer
>> TÜBİTAK BİLGEM BTE
>> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
>> T +90 262 675 3268
>> W http://www.hrzafer.com
>>
>>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by daemeon reiydelle <da...@gmail.com>.
Make virtualization an option. Federation will NOT solve your problems.
*.......*
*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Mon, Jan 26, 2015 at 1:34 AM, Azuryy Yu <az...@gmail.com> wrote:
> Hi,
>
> I think the best way is deploy HDFS federation with Hadoop 2.x.
>
> On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer <
> harun.zafer@tubitak.gov.tr> wrote:
>
>> Hi everyone,
>>
>> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
>> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
>> different versions) and ecosystem on physical machines (virtualization is
>> not an option) from different perspectives.
>>
>> As a bunch of developer we would like to work in parallel. We want every
>> team member play with his/her own cluster. However we have limited amount
>> of servers (strong machines though).
>>
>> So the question is, by changing port numbers, environment variables and
>> other configuration parameters, is it possible to setup several independent
>> clusters on same physical machines. Is there any constraints? What are the
>> possible difficulties we are to face?
>>
>> Thanks in advance
>>
>> --
>> Harun Reşit Zafer
>> TÜBİTAK BİLGEM BTE
>> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
>> T +90 262 675 3268
>> W http://www.hrzafer.com
>>
>>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
I see no possibility how federation may help to have different Clusters on _same_ machines. On top, federation isn’t production ready, since the NN can have massively issues with GC on high loaded systems, which will be the case here.
To have multiple, maybe single node, clusters the best way is to use cloud based solutions, e.g.. OpenStack with Docker containers. Also an mesos driven solution can help here, there are some good tutorials available.
BG,
Alexander
> On 26 Jan 2015, at 10:34, Azuryy Yu <az...@gmail.com> wrote:
>
> Hi,
>
> I think the best way is deploy HDFS federation with Hadoop 2.x.
>
> On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr>> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
> W http://www.hrzafer.com <http://www.hrzafer.com/>
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by daemeon reiydelle <da...@gmail.com>.
Make virtualization an option. Federation will NOT solve your problems.
*.......*
*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Mon, Jan 26, 2015 at 1:34 AM, Azuryy Yu <az...@gmail.com> wrote:
> Hi,
>
> I think the best way is deploy HDFS federation with Hadoop 2.x.
>
> On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer <
> harun.zafer@tubitak.gov.tr> wrote:
>
>> Hi everyone,
>>
>> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
>> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
>> different versions) and ecosystem on physical machines (virtualization is
>> not an option) from different perspectives.
>>
>> As a bunch of developer we would like to work in parallel. We want every
>> team member play with his/her own cluster. However we have limited amount
>> of servers (strong machines though).
>>
>> So the question is, by changing port numbers, environment variables and
>> other configuration parameters, is it possible to setup several independent
>> clusters on same physical machines. Is there any constraints? What are the
>> possible difficulties we are to face?
>>
>> Thanks in advance
>>
>> --
>> Harun Reşit Zafer
>> TÜBİTAK BİLGEM BTE
>> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
>> T +90 262 675 3268
>> W http://www.hrzafer.com
>>
>>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by daemeon reiydelle <da...@gmail.com>.
Make virtualization an option. Federation will NOT solve your problems.
*.......*
*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Mon, Jan 26, 2015 at 1:34 AM, Azuryy Yu <az...@gmail.com> wrote:
> Hi,
>
> I think the best way is deploy HDFS federation with Hadoop 2.x.
>
> On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer <
> harun.zafer@tubitak.gov.tr> wrote:
>
>> Hi everyone,
>>
>> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
>> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
>> different versions) and ecosystem on physical machines (virtualization is
>> not an option) from different perspectives.
>>
>> As a bunch of developer we would like to work in parallel. We want every
>> team member play with his/her own cluster. However we have limited amount
>> of servers (strong machines though).
>>
>> So the question is, by changing port numbers, environment variables and
>> other configuration parameters, is it possible to setup several independent
>> clusters on same physical machines. Is there any constraints? What are the
>> possible difficulties we are to face?
>>
>> Thanks in advance
>>
>> --
>> Harun Reşit Zafer
>> TÜBİTAK BİLGEM BTE
>> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
>> T +90 262 675 3268
>> W http://www.hrzafer.com
>>
>>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
I see no possibility how federation may help to have different Clusters on _same_ machines. On top, federation isn’t production ready, since the NN can have massively issues with GC on high loaded systems, which will be the case here.
To have multiple, maybe single node, clusters the best way is to use cloud based solutions, e.g.. OpenStack with Docker containers. Also an mesos driven solution can help here, there are some good tutorials available.
BG,
Alexander
> On 26 Jan 2015, at 10:34, Azuryy Yu <az...@gmail.com> wrote:
>
> Hi,
>
> I think the best way is deploy HDFS federation with Hadoop 2.x.
>
> On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr>> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
> W http://www.hrzafer.com <http://www.hrzafer.com/>
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Azuryy Yu <az...@gmail.com>.
Hi,
I think the best way is deploy HDFS federation with Hadoop 2.x.
On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer <
harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W http://www.hrzafer.com
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by daemeon reiydelle <da...@gmail.com>.
Fantastic! I was delighted to have recently worked for a large search
engine company that has moved significant components of their hadoop to to
docker containers on Ubunto, seeing amazing performance/density
improvements. And yes, the build process is really picky. Thanks SO much!
*.......*
*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Mon, Feb 2, 2015 at 3:09 AM, Hadoop Support <ha...@visolve.com>
wrote:
> Hello Ashish,
>
>
>
> Alexander reference is great. Adding to that, you can also find latest
> Hadoop docker image from below
>
>
>
> https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/
>
>
>
> Set up is as simple as that
>
> 1. Install Docker
>
> 2. Build and Pull above Image
>
> 3. Start Container (if any errors, let us know)
>
>
>
> Hope it helps!!!
>
>
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com
>
>
>
> *From:* Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
> *Sent:* Monday, February 02, 2015 3:07 PM
> *To:* user@hadoop.apache.org
> *Cc:* Harun Rešit Zafer
>
> *Subject:* Re: Multiple separate Hadoop clusters on same physical machines
>
>
>
>
> http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/
>
>
>
> Ambari based, but works quite well
>
>
>
>
>
> On 02 Feb 2015, at 10:33, Ashish Kumar9 <as...@in.ibm.com> wrote:
>
>
>
> Is there any good reference material available to follow to test docker
> and hadoop integration .
>
>
>
>
> From: <ha...@visolve.com>
> To: "'Harun Reşit Zafer'" <ha...@tubitak.gov.tr>, <
> user@hadoop.apache.org>
> Date: 02/02/2015 02:57 PM
> Subject: RE: Multiple separate Hadoop clusters on same physical
> machines
> ------------------------------
>
>
>
>
> Hello Harun,
>
> Your question is very interesting and will be useful for future Hadoop
> setups for startup/individuals too.
>
> Normally for testing purposes, we prefer you to use pseudo-distributed
> environments (i.e. installation of all cluster files in single node). You
> can refer few links which will guide you through the whole process below
> for reference:
>
>
> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
>
> Individual Pseudo Distributed Cluster Implementation:
>
> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
> http://hbase.apache.org/book.html#quickstart_pseudo
> and please check for others.
>
> From our 20 years of Server & its related Industrial experience, we
> recommend you to use VM/Instances for production & Business Critical
> environment. Other way around, if you are developing some products related
> to Hadoop, you can use docker & other related resources for development. As
> shipment to production will become stress free with the use of these tools
> with cluster environment setup.
>
> Feel free to ask for further queries.
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com
>
>
>
> *From:* Alexander Pivovarov [mailto:apivovarov@gmail.com
> <ap...@gmail.com>]
> *Sent:* Monday, February 02, 2015 12:56 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Multiple separate Hadoop clusters on same physical machines
>
> start several vms and install hadoop on each vm
> keywords: kvm, QEMU
>
> On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
> harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W http://www.hrzafer.com
>
>
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by daemeon reiydelle <da...@gmail.com>.
Fantastic! I was delighted to have recently worked for a large search
engine company that has moved significant components of their hadoop to to
docker containers on Ubunto, seeing amazing performance/density
improvements. And yes, the build process is really picky. Thanks SO much!
*.......*
*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Mon, Feb 2, 2015 at 3:09 AM, Hadoop Support <ha...@visolve.com>
wrote:
> Hello Ashish,
>
>
>
> Alexander reference is great. Adding to that, you can also find latest
> Hadoop docker image from below
>
>
>
> https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/
>
>
>
> Set up is as simple as that
>
> 1. Install Docker
>
> 2. Build and Pull above Image
>
> 3. Start Container (if any errors, let us know)
>
>
>
> Hope it helps!!!
>
>
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com
>
>
>
> *From:* Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
> *Sent:* Monday, February 02, 2015 3:07 PM
> *To:* user@hadoop.apache.org
> *Cc:* Harun Rešit Zafer
>
> *Subject:* Re: Multiple separate Hadoop clusters on same physical machines
>
>
>
>
> http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/
>
>
>
> Ambari based, but works quite well
>
>
>
>
>
> On 02 Feb 2015, at 10:33, Ashish Kumar9 <as...@in.ibm.com> wrote:
>
>
>
> Is there any good reference material available to follow to test docker
> and hadoop integration .
>
>
>
>
> From: <ha...@visolve.com>
> To: "'Harun Reşit Zafer'" <ha...@tubitak.gov.tr>, <
> user@hadoop.apache.org>
> Date: 02/02/2015 02:57 PM
> Subject: RE: Multiple separate Hadoop clusters on same physical
> machines
> ------------------------------
>
>
>
>
> Hello Harun,
>
> Your question is very interesting and will be useful for future Hadoop
> setups for startup/individuals too.
>
> Normally for testing purposes, we prefer you to use pseudo-distributed
> environments (i.e. installation of all cluster files in single node). You
> can refer few links which will guide you through the whole process below
> for reference:
>
>
> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
>
> Individual Pseudo Distributed Cluster Implementation:
>
> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
> http://hbase.apache.org/book.html#quickstart_pseudo
> and please check for others.
>
> From our 20 years of Server & its related Industrial experience, we
> recommend you to use VM/Instances for production & Business Critical
> environment. Other way around, if you are developing some products related
> to Hadoop, you can use docker & other related resources for development. As
> shipment to production will become stress free with the use of these tools
> with cluster environment setup.
>
> Feel free to ask for further queries.
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com
>
>
>
> *From:* Alexander Pivovarov [mailto:apivovarov@gmail.com
> <ap...@gmail.com>]
> *Sent:* Monday, February 02, 2015 12:56 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Multiple separate Hadoop clusters on same physical machines
>
> start several vms and install hadoop on each vm
> keywords: kvm, QEMU
>
> On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
> harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W http://www.hrzafer.com
>
>
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by daemeon reiydelle <da...@gmail.com>.
Fantastic! I was delighted to have recently worked for a large search
engine company that has moved significant components of their hadoop to to
docker containers on Ubunto, seeing amazing performance/density
improvements. And yes, the build process is really picky. Thanks SO much!
*.......*
*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Mon, Feb 2, 2015 at 3:09 AM, Hadoop Support <ha...@visolve.com>
wrote:
> Hello Ashish,
>
>
>
> Alexander reference is great. Adding to that, you can also find latest
> Hadoop docker image from below
>
>
>
> https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/
>
>
>
> Set up is as simple as that
>
> 1. Install Docker
>
> 2. Build and Pull above Image
>
> 3. Start Container (if any errors, let us know)
>
>
>
> Hope it helps!!!
>
>
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com
>
>
>
> *From:* Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
> *Sent:* Monday, February 02, 2015 3:07 PM
> *To:* user@hadoop.apache.org
> *Cc:* Harun Rešit Zafer
>
> *Subject:* Re: Multiple separate Hadoop clusters on same physical machines
>
>
>
>
> http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/
>
>
>
> Ambari based, but works quite well
>
>
>
>
>
> On 02 Feb 2015, at 10:33, Ashish Kumar9 <as...@in.ibm.com> wrote:
>
>
>
> Is there any good reference material available to follow to test docker
> and hadoop integration .
>
>
>
>
> From: <ha...@visolve.com>
> To: "'Harun Reşit Zafer'" <ha...@tubitak.gov.tr>, <
> user@hadoop.apache.org>
> Date: 02/02/2015 02:57 PM
> Subject: RE: Multiple separate Hadoop clusters on same physical
> machines
> ------------------------------
>
>
>
>
> Hello Harun,
>
> Your question is very interesting and will be useful for future Hadoop
> setups for startup/individuals too.
>
> Normally for testing purposes, we prefer you to use pseudo-distributed
> environments (i.e. installation of all cluster files in single node). You
> can refer few links which will guide you through the whole process below
> for reference:
>
>
> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
>
> Individual Pseudo Distributed Cluster Implementation:
>
> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
> http://hbase.apache.org/book.html#quickstart_pseudo
> and please check for others.
>
> From our 20 years of Server & its related Industrial experience, we
> recommend you to use VM/Instances for production & Business Critical
> environment. Other way around, if you are developing some products related
> to Hadoop, you can use docker & other related resources for development. As
> shipment to production will become stress free with the use of these tools
> with cluster environment setup.
>
> Feel free to ask for further queries.
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com
>
>
>
> *From:* Alexander Pivovarov [mailto:apivovarov@gmail.com
> <ap...@gmail.com>]
> *Sent:* Monday, February 02, 2015 12:56 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Multiple separate Hadoop clusters on same physical machines
>
> start several vms and install hadoop on each vm
> keywords: kvm, QEMU
>
> On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
> harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W http://www.hrzafer.com
>
>
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by daemeon reiydelle <da...@gmail.com>.
Fantastic! I was delighted to have recently worked for a large search
engine company that has moved significant components of their hadoop to to
docker containers on Ubunto, seeing amazing performance/density
improvements. And yes, the build process is really picky. Thanks SO much!
*.......*
*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Mon, Feb 2, 2015 at 3:09 AM, Hadoop Support <ha...@visolve.com>
wrote:
> Hello Ashish,
>
>
>
> Alexander reference is great. Adding to that, you can also find latest
> Hadoop docker image from below
>
>
>
> https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/
>
>
>
> Set up is as simple as that
>
> 1. Install Docker
>
> 2. Build and Pull above Image
>
> 3. Start Container (if any errors, let us know)
>
>
>
> Hope it helps!!!
>
>
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com
>
>
>
> *From:* Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
> *Sent:* Monday, February 02, 2015 3:07 PM
> *To:* user@hadoop.apache.org
> *Cc:* Harun Rešit Zafer
>
> *Subject:* Re: Multiple separate Hadoop clusters on same physical machines
>
>
>
>
> http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/
>
>
>
> Ambari based, but works quite well
>
>
>
>
>
> On 02 Feb 2015, at 10:33, Ashish Kumar9 <as...@in.ibm.com> wrote:
>
>
>
> Is there any good reference material available to follow to test docker
> and hadoop integration .
>
>
>
>
> From: <ha...@visolve.com>
> To: "'Harun Reşit Zafer'" <ha...@tubitak.gov.tr>, <
> user@hadoop.apache.org>
> Date: 02/02/2015 02:57 PM
> Subject: RE: Multiple separate Hadoop clusters on same physical
> machines
> ------------------------------
>
>
>
>
> Hello Harun,
>
> Your question is very interesting and will be useful for future Hadoop
> setups for startup/individuals too.
>
> Normally for testing purposes, we prefer you to use pseudo-distributed
> environments (i.e. installation of all cluster files in single node). You
> can refer few links which will guide you through the whole process below
> for reference:
>
>
> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
>
> Individual Pseudo Distributed Cluster Implementation:
>
> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
> http://hbase.apache.org/book.html#quickstart_pseudo
> and please check for others.
>
> From our 20 years of Server & its related Industrial experience, we
> recommend you to use VM/Instances for production & Business Critical
> environment. Other way around, if you are developing some products related
> to Hadoop, you can use docker & other related resources for development. As
> shipment to production will become stress free with the use of these tools
> with cluster environment setup.
>
> Feel free to ask for further queries.
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com
>
>
>
> *From:* Alexander Pivovarov [mailto:apivovarov@gmail.com
> <ap...@gmail.com>]
> *Sent:* Monday, February 02, 2015 12:56 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Multiple separate Hadoop clusters on same physical machines
>
> start several vms and install hadoop on each vm
> keywords: kvm, QEMU
>
> On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
> harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W http://www.hrzafer.com
>
>
>
>
RE: Multiple separate Hadoop clusters on same physical machines
Posted by Hadoop Support <ha...@visolve.com>.
Hello Ashish,
Alexander reference is great. Adding to that, you can also find latest Hadoop docker image from below
https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/
Set up is as simple as that
1. Install Docker
2. Build and Pull above Image
3. Start Container (if any errors, let us know)
Hope it helps!!!
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.| <http://www.visolve.com/> www.visolve.com
From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
Sent: Monday, February 02, 2015 3:07 PM
To: user@hadoop.apache.org
Cc: Harun Rešit Zafer
Subject: Re: Multiple separate Hadoop clusters on same physical machines
http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/
Ambari based, but works quite well
On 02 Feb 2015, at 10:33, Ashish Kumar9 <ashishk4@in.ibm.com <ma...@in.ibm.com> > wrote:
Is there any good reference material available to follow to test docker and hadoop integration .
From: <hadoop.support@visolve.com <ma...@visolve.com> >
To: "'Harun Reşit Zafer'" <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr> >, <user@hadoop.apache.org <ma...@hadoop.apache.org> >
Date: 02/02/2015 02:57 PM
Subject: RE: Multiple separate Hadoop clusters on same physical machines
_____
Hello Harun,
Your question is very interesting and will be useful for future Hadoop setups for startup/individuals too.
Normally for testing purposes, we prefer you to use pseudo-distributed environments (i.e. installation of all cluster files in single node). You can refer few links which will guide you through the whole process below for reference:
<https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
Individual Pseudo Distributed Cluster Implementation:
<http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
<http://hbase.apache.org/book.html#quickstart_pseudo> http://hbase.apache.org/book.html#quickstart_pseudo
and please check for others.
>From our 20 years of Server & its related Industrial experience, we recommend you to use VM/Instances for production & Business Critical environment. Other way around, if you are developing some products related to Hadoop, you can use docker & other related resources for development. As shipment to production will become stress free with the use of these tools with cluster environment setup.
Feel free to ask for further queries.
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.| <http://www.visolve.com/> www.visolve.com
From: Alexander Pivovarov [ <ma...@gmail.com> mailto:apivovarov@gmail.com]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.org <ma...@hadoop.apache.org>
Subject: Re: Multiple separate Hadoop clusters on same physical machines
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr> > wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
W http://www.hrzafer.com <http://www.hrzafer.com/>
RE: Multiple separate Hadoop clusters on same physical machines
Posted by Hadoop Support <ha...@visolve.com>.
Hello Ashish,
Alexander reference is great. Adding to that, you can also find latest Hadoop docker image from below
https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/
Set up is as simple as that
1. Install Docker
2. Build and Pull above Image
3. Start Container (if any errors, let us know)
Hope it helps!!!
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.| <http://www.visolve.com/> www.visolve.com
From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
Sent: Monday, February 02, 2015 3:07 PM
To: user@hadoop.apache.org
Cc: Harun Rešit Zafer
Subject: Re: Multiple separate Hadoop clusters on same physical machines
http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/
Ambari based, but works quite well
On 02 Feb 2015, at 10:33, Ashish Kumar9 <ashishk4@in.ibm.com <ma...@in.ibm.com> > wrote:
Is there any good reference material available to follow to test docker and hadoop integration .
From: <hadoop.support@visolve.com <ma...@visolve.com> >
To: "'Harun Reşit Zafer'" <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr> >, <user@hadoop.apache.org <ma...@hadoop.apache.org> >
Date: 02/02/2015 02:57 PM
Subject: RE: Multiple separate Hadoop clusters on same physical machines
_____
Hello Harun,
Your question is very interesting and will be useful for future Hadoop setups for startup/individuals too.
Normally for testing purposes, we prefer you to use pseudo-distributed environments (i.e. installation of all cluster files in single node). You can refer few links which will guide you through the whole process below for reference:
<https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
Individual Pseudo Distributed Cluster Implementation:
<http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
<http://hbase.apache.org/book.html#quickstart_pseudo> http://hbase.apache.org/book.html#quickstart_pseudo
and please check for others.
>From our 20 years of Server & its related Industrial experience, we recommend you to use VM/Instances for production & Business Critical environment. Other way around, if you are developing some products related to Hadoop, you can use docker & other related resources for development. As shipment to production will become stress free with the use of these tools with cluster environment setup.
Feel free to ask for further queries.
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.| <http://www.visolve.com/> www.visolve.com
From: Alexander Pivovarov [ <ma...@gmail.com> mailto:apivovarov@gmail.com]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.org <ma...@hadoop.apache.org>
Subject: Re: Multiple separate Hadoop clusters on same physical machines
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr> > wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
W http://www.hrzafer.com <http://www.hrzafer.com/>
RE: Multiple separate Hadoop clusters on same physical machines
Posted by Hadoop Support <ha...@visolve.com>.
Hello Ashish,
Alexander reference is great. Adding to that, you can also find latest Hadoop docker image from below
https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/
Set up is as simple as that
1. Install Docker
2. Build and Pull above Image
3. Start Container (if any errors, let us know)
Hope it helps!!!
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.| <http://www.visolve.com/> www.visolve.com
From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
Sent: Monday, February 02, 2015 3:07 PM
To: user@hadoop.apache.org
Cc: Harun Rešit Zafer
Subject: Re: Multiple separate Hadoop clusters on same physical machines
http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/
Ambari based, but works quite well
On 02 Feb 2015, at 10:33, Ashish Kumar9 <ashishk4@in.ibm.com <ma...@in.ibm.com> > wrote:
Is there any good reference material available to follow to test docker and hadoop integration .
From: <hadoop.support@visolve.com <ma...@visolve.com> >
To: "'Harun Reşit Zafer'" <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr> >, <user@hadoop.apache.org <ma...@hadoop.apache.org> >
Date: 02/02/2015 02:57 PM
Subject: RE: Multiple separate Hadoop clusters on same physical machines
_____
Hello Harun,
Your question is very interesting and will be useful for future Hadoop setups for startup/individuals too.
Normally for testing purposes, we prefer you to use pseudo-distributed environments (i.e. installation of all cluster files in single node). You can refer few links which will guide you through the whole process below for reference:
<https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
Individual Pseudo Distributed Cluster Implementation:
<http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
<http://hbase.apache.org/book.html#quickstart_pseudo> http://hbase.apache.org/book.html#quickstart_pseudo
and please check for others.
>From our 20 years of Server & its related Industrial experience, we recommend you to use VM/Instances for production & Business Critical environment. Other way around, if you are developing some products related to Hadoop, you can use docker & other related resources for development. As shipment to production will become stress free with the use of these tools with cluster environment setup.
Feel free to ask for further queries.
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.| <http://www.visolve.com/> www.visolve.com
From: Alexander Pivovarov [ <ma...@gmail.com> mailto:apivovarov@gmail.com]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.org <ma...@hadoop.apache.org>
Subject: Re: Multiple separate Hadoop clusters on same physical machines
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr> > wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
W http://www.hrzafer.com <http://www.hrzafer.com/>
RE: Multiple separate Hadoop clusters on same physical machines
Posted by Hadoop Support <ha...@visolve.com>.
Hello Ashish,
Alexander reference is great. Adding to that, you can also find latest Hadoop docker image from below
https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/
Set up is as simple as that
1. Install Docker
2. Build and Pull above Image
3. Start Container (if any errors, let us know)
Hope it helps!!!
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.| <http://www.visolve.com/> www.visolve.com
From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
Sent: Monday, February 02, 2015 3:07 PM
To: user@hadoop.apache.org
Cc: Harun Rešit Zafer
Subject: Re: Multiple separate Hadoop clusters on same physical machines
http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/
Ambari based, but works quite well
On 02 Feb 2015, at 10:33, Ashish Kumar9 <ashishk4@in.ibm.com <ma...@in.ibm.com> > wrote:
Is there any good reference material available to follow to test docker and hadoop integration .
From: <hadoop.support@visolve.com <ma...@visolve.com> >
To: "'Harun Reşit Zafer'" <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr> >, <user@hadoop.apache.org <ma...@hadoop.apache.org> >
Date: 02/02/2015 02:57 PM
Subject: RE: Multiple separate Hadoop clusters on same physical machines
_____
Hello Harun,
Your question is very interesting and will be useful for future Hadoop setups for startup/individuals too.
Normally for testing purposes, we prefer you to use pseudo-distributed environments (i.e. installation of all cluster files in single node). You can refer few links which will guide you through the whole process below for reference:
<https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
Individual Pseudo Distributed Cluster Implementation:
<http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
<http://hbase.apache.org/book.html#quickstart_pseudo> http://hbase.apache.org/book.html#quickstart_pseudo
and please check for others.
>From our 20 years of Server & its related Industrial experience, we recommend you to use VM/Instances for production & Business Critical environment. Other way around, if you are developing some products related to Hadoop, you can use docker & other related resources for development. As shipment to production will become stress free with the use of these tools with cluster environment setup.
Feel free to ask for further queries.
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.| <http://www.visolve.com/> www.visolve.com
From: Alexander Pivovarov [ <ma...@gmail.com> mailto:apivovarov@gmail.com]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.org <ma...@hadoop.apache.org>
Subject: Re: Multiple separate Hadoop clusters on same physical machines
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr> > wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
W http://www.hrzafer.com <http://www.hrzafer.com/>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/ <http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/>
Ambari based, but works quite well
> On 02 Feb 2015, at 10:33, Ashish Kumar9 <as...@in.ibm.com> wrote:
>
> Is there any good reference material available to follow to test docker and hadoop integration .
>
>
>
>
> From: <ha...@visolve.com>
> To: "'Harun Reşit Zafer'" <ha...@tubitak.gov.tr>, <us...@hadoop.apache.org>
> Date: 02/02/2015 02:57 PM
> Subject: RE: Multiple separate Hadoop clusters on same physical machines
>
>
>
> Hello Harun,
>
> Your question is very interesting and will be useful for future Hadoop setups for startup/individuals too.
>
> Normally for testing purposes, we prefer you to use pseudo-distributed environments (i.e. installation of all cluster files in single node). You can refer few links which will guide you through the whole process below for reference:
>
> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment <https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment>
>
> Individual Pseudo Distributed Cluster Implementation:
>
> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/ <http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/>
> http://hbase.apache.org/book.html#quickstart_pseudo <http://hbase.apache.org/book.html#quickstart_pseudo>
> and please check for others.
>
> From our 20 years of Server & its related Industrial experience, we recommend you to use VM/Instances for production & Business Critical environment. Other way around, if you are developing some products related to Hadoop, you can use docker & other related resources for development. As shipment to production will become stress free with the use of these tools with cluster environment setup.
>
> Feel free to ask for further queries.
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com <http://www.visolve.com/>
>
>
>
> From: Alexander Pivovarov [mailto:apivovarov@gmail.com <ma...@gmail.com>]
> Sent: Monday, February 02, 2015 12:56 PM
> To: user@hadoop.apache.org
> Subject: Re: Multiple separate Hadoop clusters on same physical machines
>
> start several vms and install hadoop on each vm
> keywords: kvm, QEMU
>
> On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr>> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
> W http://www.hrzafer.com <http://www.hrzafer.com/>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/ <http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/>
Ambari based, but works quite well
> On 02 Feb 2015, at 10:33, Ashish Kumar9 <as...@in.ibm.com> wrote:
>
> Is there any good reference material available to follow to test docker and hadoop integration .
>
>
>
>
> From: <ha...@visolve.com>
> To: "'Harun Reşit Zafer'" <ha...@tubitak.gov.tr>, <us...@hadoop.apache.org>
> Date: 02/02/2015 02:57 PM
> Subject: RE: Multiple separate Hadoop clusters on same physical machines
>
>
>
> Hello Harun,
>
> Your question is very interesting and will be useful for future Hadoop setups for startup/individuals too.
>
> Normally for testing purposes, we prefer you to use pseudo-distributed environments (i.e. installation of all cluster files in single node). You can refer few links which will guide you through the whole process below for reference:
>
> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment <https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment>
>
> Individual Pseudo Distributed Cluster Implementation:
>
> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/ <http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/>
> http://hbase.apache.org/book.html#quickstart_pseudo <http://hbase.apache.org/book.html#quickstart_pseudo>
> and please check for others.
>
> From our 20 years of Server & its related Industrial experience, we recommend you to use VM/Instances for production & Business Critical environment. Other way around, if you are developing some products related to Hadoop, you can use docker & other related resources for development. As shipment to production will become stress free with the use of these tools with cluster environment setup.
>
> Feel free to ask for further queries.
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com <http://www.visolve.com/>
>
>
>
> From: Alexander Pivovarov [mailto:apivovarov@gmail.com <ma...@gmail.com>]
> Sent: Monday, February 02, 2015 12:56 PM
> To: user@hadoop.apache.org
> Subject: Re: Multiple separate Hadoop clusters on same physical machines
>
> start several vms and install hadoop on each vm
> keywords: kvm, QEMU
>
> On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr>> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
> W http://www.hrzafer.com <http://www.hrzafer.com/>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/ <http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/>
Ambari based, but works quite well
> On 02 Feb 2015, at 10:33, Ashish Kumar9 <as...@in.ibm.com> wrote:
>
> Is there any good reference material available to follow to test docker and hadoop integration .
>
>
>
>
> From: <ha...@visolve.com>
> To: "'Harun Reşit Zafer'" <ha...@tubitak.gov.tr>, <us...@hadoop.apache.org>
> Date: 02/02/2015 02:57 PM
> Subject: RE: Multiple separate Hadoop clusters on same physical machines
>
>
>
> Hello Harun,
>
> Your question is very interesting and will be useful for future Hadoop setups for startup/individuals too.
>
> Normally for testing purposes, we prefer you to use pseudo-distributed environments (i.e. installation of all cluster files in single node). You can refer few links which will guide you through the whole process below for reference:
>
> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment <https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment>
>
> Individual Pseudo Distributed Cluster Implementation:
>
> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/ <http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/>
> http://hbase.apache.org/book.html#quickstart_pseudo <http://hbase.apache.org/book.html#quickstart_pseudo>
> and please check for others.
>
> From our 20 years of Server & its related Industrial experience, we recommend you to use VM/Instances for production & Business Critical environment. Other way around, if you are developing some products related to Hadoop, you can use docker & other related resources for development. As shipment to production will become stress free with the use of these tools with cluster environment setup.
>
> Feel free to ask for further queries.
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com <http://www.visolve.com/>
>
>
>
> From: Alexander Pivovarov [mailto:apivovarov@gmail.com <ma...@gmail.com>]
> Sent: Monday, February 02, 2015 12:56 PM
> To: user@hadoop.apache.org
> Subject: Re: Multiple separate Hadoop clusters on same physical machines
>
> start several vms and install hadoop on each vm
> keywords: kvm, QEMU
>
> On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr>> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
> W http://www.hrzafer.com <http://www.hrzafer.com/>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/ <http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/>
Ambari based, but works quite well
> On 02 Feb 2015, at 10:33, Ashish Kumar9 <as...@in.ibm.com> wrote:
>
> Is there any good reference material available to follow to test docker and hadoop integration .
>
>
>
>
> From: <ha...@visolve.com>
> To: "'Harun Reşit Zafer'" <ha...@tubitak.gov.tr>, <us...@hadoop.apache.org>
> Date: 02/02/2015 02:57 PM
> Subject: RE: Multiple separate Hadoop clusters on same physical machines
>
>
>
> Hello Harun,
>
> Your question is very interesting and will be useful for future Hadoop setups for startup/individuals too.
>
> Normally for testing purposes, we prefer you to use pseudo-distributed environments (i.e. installation of all cluster files in single node). You can refer few links which will guide you through the whole process below for reference:
>
> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment <https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment>
>
> Individual Pseudo Distributed Cluster Implementation:
>
> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/ <http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/>
> http://hbase.apache.org/book.html#quickstart_pseudo <http://hbase.apache.org/book.html#quickstart_pseudo>
> and please check for others.
>
> From our 20 years of Server & its related Industrial experience, we recommend you to use VM/Instances for production & Business Critical environment. Other way around, if you are developing some products related to Hadoop, you can use docker & other related resources for development. As shipment to production will become stress free with the use of these tools with cluster environment setup.
>
> Feel free to ask for further queries.
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com <http://www.visolve.com/>
>
>
>
> From: Alexander Pivovarov [mailto:apivovarov@gmail.com <ma...@gmail.com>]
> Sent: Monday, February 02, 2015 12:56 PM
> To: user@hadoop.apache.org
> Subject: Re: Multiple separate Hadoop clusters on same physical machines
>
> start several vms and install hadoop on each vm
> keywords: kvm, QEMU
>
> On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr>> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
> W http://www.hrzafer.com <http://www.hrzafer.com/>
>
RE: Multiple separate Hadoop clusters on same physical machines
Posted by Ashish Kumar9 <as...@in.ibm.com>.
Is there any good reference material available to follow to test docker
and hadoop integration .
From: <ha...@visolve.com>
To: "'Harun Reşit Zafer'" <ha...@tubitak.gov.tr>,
<us...@hadoop.apache.org>
Date: 02/02/2015 02:57 PM
Subject: RE: Multiple separate Hadoop clusters on same physical
machines
Hello Harun,
Your question is very interesting and will be useful for future Hadoop
setups for startup/individuals too.
Normally for testing purposes, we prefer you to use pseudo-distributed
environments (i.e. installation of all cluster files in single node). You
can refer few links which will guide you through the whole process below
for reference:
https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
Individual Pseudo Distributed Cluster Implementation:
http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
http://hbase.apache.org/book.html#quickstart_pseudo
and please check for others.
>From our 20 years of Server & its related Industrial experience, we
recommend you to use VM/Instances for production & Business Critical
environment. Other way around, if you are developing some products related
to Hadoop, you can use docker & other related resources for development.
As shipment to production will become stress free with the use of these
tools with cluster environment setup.
Feel free to ask for further queries.
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.|www.visolve.com
From: Alexander Pivovarov [mailto:apivovarov@gmail.com]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.org
Subject: Re: Multiple separate Hadoop clusters on same physical machines
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
harun.zafer@tubitak.gov.tr> wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
different versions) and ecosystem on physical machines (virtualization is
not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every
team member play with his/her own cluster. However we have limited amount
of servers (strong machines though).
So the question is, by changing port numbers, environment variables and
other configuration parameters, is it possible to setup several
independent clusters on same physical machines. Is there any constraints?
What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268
W http://www.hrzafer.com
RE: Multiple separate Hadoop clusters on same physical machines
Posted by Ashish Kumar9 <as...@in.ibm.com>.
Is there any good reference material available to follow to test docker
and hadoop integration .
From: <ha...@visolve.com>
To: "'Harun Reşit Zafer'" <ha...@tubitak.gov.tr>,
<us...@hadoop.apache.org>
Date: 02/02/2015 02:57 PM
Subject: RE: Multiple separate Hadoop clusters on same physical
machines
Hello Harun,
Your question is very interesting and will be useful for future Hadoop
setups for startup/individuals too.
Normally for testing purposes, we prefer you to use pseudo-distributed
environments (i.e. installation of all cluster files in single node). You
can refer few links which will guide you through the whole process below
for reference:
https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
Individual Pseudo Distributed Cluster Implementation:
http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
http://hbase.apache.org/book.html#quickstart_pseudo
and please check for others.
>From our 20 years of Server & its related Industrial experience, we
recommend you to use VM/Instances for production & Business Critical
environment. Other way around, if you are developing some products related
to Hadoop, you can use docker & other related resources for development.
As shipment to production will become stress free with the use of these
tools with cluster environment setup.
Feel free to ask for further queries.
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.|www.visolve.com
From: Alexander Pivovarov [mailto:apivovarov@gmail.com]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.org
Subject: Re: Multiple separate Hadoop clusters on same physical machines
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
harun.zafer@tubitak.gov.tr> wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
different versions) and ecosystem on physical machines (virtualization is
not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every
team member play with his/her own cluster. However we have limited amount
of servers (strong machines though).
So the question is, by changing port numbers, environment variables and
other configuration parameters, is it possible to setup several
independent clusters on same physical machines. Is there any constraints?
What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268
W http://www.hrzafer.com
RE: Multiple separate Hadoop clusters on same physical machines
Posted by Ashish Kumar9 <as...@in.ibm.com>.
Is there any good reference material available to follow to test docker
and hadoop integration .
From: <ha...@visolve.com>
To: "'Harun Reşit Zafer'" <ha...@tubitak.gov.tr>,
<us...@hadoop.apache.org>
Date: 02/02/2015 02:57 PM
Subject: RE: Multiple separate Hadoop clusters on same physical
machines
Hello Harun,
Your question is very interesting and will be useful for future Hadoop
setups for startup/individuals too.
Normally for testing purposes, we prefer you to use pseudo-distributed
environments (i.e. installation of all cluster files in single node). You
can refer few links which will guide you through the whole process below
for reference:
https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
Individual Pseudo Distributed Cluster Implementation:
http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
http://hbase.apache.org/book.html#quickstart_pseudo
and please check for others.
>From our 20 years of Server & its related Industrial experience, we
recommend you to use VM/Instances for production & Business Critical
environment. Other way around, if you are developing some products related
to Hadoop, you can use docker & other related resources for development.
As shipment to production will become stress free with the use of these
tools with cluster environment setup.
Feel free to ask for further queries.
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.|www.visolve.com
From: Alexander Pivovarov [mailto:apivovarov@gmail.com]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.org
Subject: Re: Multiple separate Hadoop clusters on same physical machines
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
harun.zafer@tubitak.gov.tr> wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
different versions) and ecosystem on physical machines (virtualization is
not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every
team member play with his/her own cluster. However we have limited amount
of servers (strong machines though).
So the question is, by changing port numbers, environment variables and
other configuration parameters, is it possible to setup several
independent clusters on same physical machines. Is there any constraints?
What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268
W http://www.hrzafer.com
RE: Multiple separate Hadoop clusters on same physical machines
Posted by Ashish Kumar9 <as...@in.ibm.com>.
Is there any good reference material available to follow to test docker
and hadoop integration .
From: <ha...@visolve.com>
To: "'Harun Reşit Zafer'" <ha...@tubitak.gov.tr>,
<us...@hadoop.apache.org>
Date: 02/02/2015 02:57 PM
Subject: RE: Multiple separate Hadoop clusters on same physical
machines
Hello Harun,
Your question is very interesting and will be useful for future Hadoop
setups for startup/individuals too.
Normally for testing purposes, we prefer you to use pseudo-distributed
environments (i.e. installation of all cluster files in single node). You
can refer few links which will guide you through the whole process below
for reference:
https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
Individual Pseudo Distributed Cluster Implementation:
http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
http://hbase.apache.org/book.html#quickstart_pseudo
and please check for others.
>From our 20 years of Server & its related Industrial experience, we
recommend you to use VM/Instances for production & Business Critical
environment. Other way around, if you are developing some products related
to Hadoop, you can use docker & other related resources for development.
As shipment to production will become stress free with the use of these
tools with cluster environment setup.
Feel free to ask for further queries.
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.|www.visolve.com
From: Alexander Pivovarov [mailto:apivovarov@gmail.com]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.org
Subject: Re: Multiple separate Hadoop clusters on same physical machines
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
harun.zafer@tubitak.gov.tr> wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
different versions) and ecosystem on physical machines (virtualization is
not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every
team member play with his/her own cluster. However we have limited amount
of servers (strong machines though).
So the question is, by changing port numbers, environment variables and
other configuration parameters, is it possible to setup several
independent clusters on same physical machines. Is there any constraints?
What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268
W http://www.hrzafer.com
RE: Multiple separate Hadoop clusters on same physical machines
Posted by ha...@visolve.com.
Hello Harun,
Your question is very interesting and will be useful for future Hadoop setups for startup/individuals too.
Normally for testing purposes, we prefer you to use pseudo-distributed environments (i.e. installation of all cluster files in single node). You can refer few links which will guide you through the whole process below for reference:
https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
Individual Pseudo Distributed Cluster Implementation:
http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
http://hbase.apache.org/book.html#quickstart_pseudo
and please check for others.
>From our 20 years of Server & its related Industrial experience, we recommend you to use VM/Instances for production & Business Critical environment. Other way around, if you are developing some products related to Hadoop, you can use docker & other related resources for development. As shipment to production will become stress free with the use of these tools with cluster environment setup.
Feel free to ask for further queries.
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.| <http://www.visolve.com> www.visolve.com
From: Alexander Pivovarov [mailto:apivovarov@gmail.com]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.org
Subject: Re: Multiple separate Hadoop clusters on same physical machines
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr> > wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
W http://www.hrzafer.com
RE: Multiple separate Hadoop clusters on same physical machines
Posted by ha...@visolve.com.
Hello Harun,
Your question is very interesting and will be useful for future Hadoop setups for startup/individuals too.
Normally for testing purposes, we prefer you to use pseudo-distributed environments (i.e. installation of all cluster files in single node). You can refer few links which will guide you through the whole process below for reference:
https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
Individual Pseudo Distributed Cluster Implementation:
http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
http://hbase.apache.org/book.html#quickstart_pseudo
and please check for others.
>From our 20 years of Server & its related Industrial experience, we recommend you to use VM/Instances for production & Business Critical environment. Other way around, if you are developing some products related to Hadoop, you can use docker & other related resources for development. As shipment to production will become stress free with the use of these tools with cluster environment setup.
Feel free to ask for further queries.
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.| <http://www.visolve.com> www.visolve.com
From: Alexander Pivovarov [mailto:apivovarov@gmail.com]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.org
Subject: Re: Multiple separate Hadoop clusters on same physical machines
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr> > wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
W http://www.hrzafer.com
RE: Multiple separate Hadoop clusters on same physical machines
Posted by ha...@visolve.com.
Hello Harun,
Your question is very interesting and will be useful for future Hadoop setups for startup/individuals too.
Normally for testing purposes, we prefer you to use pseudo-distributed environments (i.e. installation of all cluster files in single node). You can refer few links which will guide you through the whole process below for reference:
https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
Individual Pseudo Distributed Cluster Implementation:
http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
http://hbase.apache.org/book.html#quickstart_pseudo
and please check for others.
>From our 20 years of Server & its related Industrial experience, we recommend you to use VM/Instances for production & Business Critical environment. Other way around, if you are developing some products related to Hadoop, you can use docker & other related resources for development. As shipment to production will become stress free with the use of these tools with cluster environment setup.
Feel free to ask for further queries.
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.| <http://www.visolve.com> www.visolve.com
From: Alexander Pivovarov [mailto:apivovarov@gmail.com]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.org
Subject: Re: Multiple separate Hadoop clusters on same physical machines
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr> > wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
W http://www.hrzafer.com
RE: Multiple separate Hadoop clusters on same physical machines
Posted by ha...@visolve.com.
Hello Harun,
Your question is very interesting and will be useful for future Hadoop setups for startup/individuals too.
Normally for testing purposes, we prefer you to use pseudo-distributed environments (i.e. installation of all cluster files in single node). You can refer few links which will guide you through the whole process below for reference:
https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
Individual Pseudo Distributed Cluster Implementation:
http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
http://hbase.apache.org/book.html#quickstart_pseudo
and please check for others.
>From our 20 years of Server & its related Industrial experience, we recommend you to use VM/Instances for production & Business Critical environment. Other way around, if you are developing some products related to Hadoop, you can use docker & other related resources for development. As shipment to production will become stress free with the use of these tools with cluster environment setup.
Feel free to ask for further queries.
Thanks and Regards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.| <http://www.visolve.com> www.visolve.com
From: Alexander Pivovarov [mailto:apivovarov@gmail.com]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.org
Subject: Re: Multiple separate Hadoop clusters on same physical machines
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <harun.zafer@tubitak.gov.tr <ma...@tubitak.gov.tr> > wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe different versions) and ecosystem on physical machines (virtualization is not an option) from different perspectives.
As a bunch of developer we would like to work in parallel. We want every team member play with his/her own cluster. However we have limited amount of servers (strong machines though).
So the question is, by changing port numbers, environment variables and other configuration parameters, is it possible to setup several independent clusters on same physical machines. Is there any constraints? What are the possible difficulties we are to face?
Thanks in advance
--
Harun Reşit Zafer
TÜBİTAK BİLGEM BTE
Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
T +90 262 675 3268 <tel:%2B90%20262%20675%203268>
W http://www.hrzafer.com
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Alexander Pivovarov <ap...@gmail.com>.
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W http://www.hrzafer.com
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Azuryy Yu <az...@gmail.com>.
Hi,
I think the best way is deploy HDFS federation with Hadoop 2.x.
On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer <
harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W http://www.hrzafer.com
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Azuryy Yu <az...@gmail.com>.
Hi,
I think the best way is deploy HDFS federation with Hadoop 2.x.
On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer <
harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W http://www.hrzafer.com
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Alexander Pivovarov <ap...@gmail.com>.
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W http://www.hrzafer.com
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Azuryy Yu <az...@gmail.com>.
Hi,
I think the best way is deploy HDFS federation with Hadoop 2.x.
On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer <
harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W http://www.hrzafer.com
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Alexander Pivovarov <ap...@gmail.com>.
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W http://www.hrzafer.com
>
>
Re: Multiple separate Hadoop clusters on same physical machines
Posted by Alexander Pivovarov <ap...@gmail.com>.
start several vms and install hadoop on each vm
keywords: kvm, QEMU
On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W http://www.hrzafer.com
>
>