You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Sasaki Kai <le...@me.com> on 2016/11/09 13:38:55 UTC

Multiple node Hadoop cluster on Docker for test and debugging

Hi Hadoop developers

The other day I created a tool for launching multiple node hadoop cluster on docker container.
You can easily launch multiple node hadoop cluster from your Hadoop source code. 
It is useful for testing and debugging. Actually I often use it before submitting a patch to Hadoop project.
https://github.com/Lewuathe/docker-hadoop-cluster <https://github.com/Lewuathe/docker-hadoop-cluster>

And I also updated to build the latest trunk image automatically and upload onto Docker Hub.
So you can easily check and test the latest trunk branch in the environment which is more close to actual usage.

If you already installed docker and docker-compose, what needed is docker-compose.yml like this.

version: '2'

services:
  master:
    image: lewuathe/hadoop-master
    ports:
      - "9870:9870"
      - "8088:8088"
      - "19888:19888"
      - "8188:8188"
    container_name: "master"
  slave1:
    image: lewuathe/hadoop-slave
    container_name: "slave1"
    depends_on:
      - master
    ports:
      - "9901:9864"
      - "8041:8042"
  slave2:
    image: lewuathe/hadoop-slave
    container_name: "slave2"
    depends_on:
      - master
    ports:
      - "9902:9864"
      - "8042:8042"

The usage in detail is described in the repository.
https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md <https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md>

I would be glad if you use this tool for developing and debugging and make our development more efficient.
Please give me any feedbacks to me. Thanks you!

————————
Kai Sasaki
mail: lewuathe@me.com <ma...@me.com>
github: https://github.com/Lewuathe <https://github.com/Lewuathe>

Re: Multiple node Hadoop cluster on Docker for test and debugging

Posted by Sasaki Kai <le...@me.com>.
Roman, Zhe

Thanks for the response!

I was not familiar with Apache BigTop. I also try to use it. Thanks!

Kai Sasaki

> On 2016 Nov 10, at 9:07, Roman Shaposhnik <rv...@apache.org> wrote:
> 
> On Wed, Nov 9, 2016 at 9:30 AM, Zhe Zhang <zh...@gmail.com> wrote:
>> Nice work Kai! Quick comment: have you considered Bigtop? It allows
>> creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
>> a custom built Hadoop (from the instructions I read, it builds from a
>> certain git branch; I'm not sure if that customizable).
> 
> That's perfectly customizable. What's cool about Bigtop is that it allows
> you to provision not just Hadoop but tons of other big data projects since
> it is all driven by the set of Puppet recipes. The very same set of Puppet
> recipes that also drive Canonical's Juju:
>    https://jujucharms.com/big-data
> 
> Thanks,
> Roman.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: Multiple node Hadoop cluster on Docker for test and debugging

Posted by Sasaki Kai <le...@me.com>.
Roman, Zhe

Thanks for the response!

I was not familiar with Apache BigTop. I also try to use it. Thanks!

Kai Sasaki

> On 2016 Nov 10, at 9:07, Roman Shaposhnik <rv...@apache.org> wrote:
> 
> On Wed, Nov 9, 2016 at 9:30 AM, Zhe Zhang <zh...@gmail.com> wrote:
>> Nice work Kai! Quick comment: have you considered Bigtop? It allows
>> creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
>> a custom built Hadoop (from the instructions I read, it builds from a
>> certain git branch; I'm not sure if that customizable).
> 
> That's perfectly customizable. What's cool about Bigtop is that it allows
> you to provision not just Hadoop but tons of other big data projects since
> it is all driven by the set of Puppet recipes. The very same set of Puppet
> recipes that also drive Canonical's Juju:
>    https://jujucharms.com/big-data
> 
> Thanks,
> Roman.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: Multiple node Hadoop cluster on Docker for test and debugging

Posted by Sasaki Kai <le...@me.com>.
Roman, Zhe

Thanks for the response!

I was not familiar with Apache BigTop. I also try to use it. Thanks!

Kai Sasaki

> On 2016 Nov 10, at 9:07, Roman Shaposhnik <rv...@apache.org> wrote:
> 
> On Wed, Nov 9, 2016 at 9:30 AM, Zhe Zhang <zh...@gmail.com> wrote:
>> Nice work Kai! Quick comment: have you considered Bigtop? It allows
>> creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
>> a custom built Hadoop (from the instructions I read, it builds from a
>> certain git branch; I'm not sure if that customizable).
> 
> That's perfectly customizable. What's cool about Bigtop is that it allows
> you to provision not just Hadoop but tons of other big data projects since
> it is all driven by the set of Puppet recipes. The very same set of Puppet
> recipes that also drive Canonical's Juju:
>    https://jujucharms.com/big-data
> 
> Thanks,
> Roman.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: Multiple node Hadoop cluster on Docker for test and debugging

Posted by Roman Shaposhnik <rv...@apache.org>.
On Wed, Nov 9, 2016 at 9:30 AM, Zhe Zhang <zh...@gmail.com> wrote:
> Nice work Kai! Quick comment: have you considered Bigtop? It allows
> creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
> a custom built Hadoop (from the instructions I read, it builds from a
> certain git branch; I'm not sure if that customizable).

That's perfectly customizable. What's cool about Bigtop is that it allows
you to provision not just Hadoop but tons of other big data projects since
it is all driven by the set of Puppet recipes. The very same set of Puppet
recipes that also drive Canonical's Juju:
    https://jujucharms.com/big-data

Thanks,
Roman.

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: Multiple node Hadoop cluster on Docker for test and debugging

Posted by Roman Shaposhnik <rv...@apache.org>.
On Wed, Nov 9, 2016 at 9:30 AM, Zhe Zhang <zh...@gmail.com> wrote:
> Nice work Kai! Quick comment: have you considered Bigtop? It allows
> creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
> a custom built Hadoop (from the instructions I read, it builds from a
> certain git branch; I'm not sure if that customizable).

That's perfectly customizable. What's cool about Bigtop is that it allows
you to provision not just Hadoop but tons of other big data projects since
it is all driven by the set of Puppet recipes. The very same set of Puppet
recipes that also drive Canonical's Juju:
    https://jujucharms.com/big-data

Thanks,
Roman.

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: Multiple node Hadoop cluster on Docker for test and debugging

Posted by Roman Shaposhnik <rv...@apache.org>.
On Wed, Nov 9, 2016 at 9:30 AM, Zhe Zhang <zh...@gmail.com> wrote:
> Nice work Kai! Quick comment: have you considered Bigtop? It allows
> creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
> a custom built Hadoop (from the instructions I read, it builds from a
> certain git branch; I'm not sure if that customizable).

That's perfectly customizable. What's cool about Bigtop is that it allows
you to provision not just Hadoop but tons of other big data projects since
it is all driven by the set of Puppet recipes. The very same set of Puppet
recipes that also drive Canonical's Juju:
    https://jujucharms.com/big-data

Thanks,
Roman.

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: Multiple node Hadoop cluster on Docker for test and debugging

Posted by Zhe Zhang <zh...@gmail.com>.
Nice work Kai! Quick comment: have you considered Bigtop? It allows
creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
a custom built Hadoop (from the instructions I read, it builds from a
certain git branch; I'm not sure if that customizable).

On Wed, Nov 9, 2016 at 5:39 AM Sasaki Kai <le...@me.com> wrote:

> Hi Hadoop developers
>
> The other day I created a tool for launching multiple node hadoop cluster
> on docker container.
> You can easily launch multiple node hadoop cluster from your Hadoop source
> code.
> It is useful for testing and debugging. Actually I often use it before
> submitting a patch to Hadoop project.
> https://github.com/Lewuathe/docker-hadoop-cluster <
> https://github.com/Lewuathe/docker-hadoop-cluster>
>
> And I also updated to build the latest trunk image automatically and
> upload onto Docker Hub.
> So you can easily check and test the latest trunk branch in the
> environment which is more close to actual usage.
>
> If you already installed docker and docker-compose, what needed is
> docker-compose.yml like this.
>
> version: '2'
>
> services:
>   master:
>     image: lewuathe/hadoop-master
>     ports:
>       - "9870:9870"
>       - "8088:8088"
>       - "19888:19888"
>       - "8188:8188"
>     container_name: "master"
>   slave1:
>     image: lewuathe/hadoop-slave
>     container_name: "slave1"
>     depends_on:
>       - master
>     ports:
>       - "9901:9864"
>       - "8041:8042"
>   slave2:
>     image: lewuathe/hadoop-slave
>     container_name: "slave2"
>     depends_on:
>       - master
>     ports:
>       - "9902:9864"
>       - "8042:8042"
>
> The usage in detail is described in the repository.
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md <
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md>
>
> I would be glad if you use this tool for developing and debugging and make
> our development more efficient.
> Please give me any feedbacks to me. Thanks you!
>
> ————————
> Kai Sasaki
> mail: lewuathe@me.com <ma...@me.com>
> github: https://github.com/Lewuathe <https://github.com/Lewuathe>
>
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: Multiple node Hadoop cluster on Docker for test and debugging

Posted by Zhe Zhang <zh...@gmail.com>.
Nice work Kai! Quick comment: have you considered Bigtop? It allows
creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
a custom built Hadoop (from the instructions I read, it builds from a
certain git branch; I'm not sure if that customizable).

On Wed, Nov 9, 2016 at 5:39 AM Sasaki Kai <le...@me.com> wrote:

> Hi Hadoop developers
>
> The other day I created a tool for launching multiple node hadoop cluster
> on docker container.
> You can easily launch multiple node hadoop cluster from your Hadoop source
> code.
> It is useful for testing and debugging. Actually I often use it before
> submitting a patch to Hadoop project.
> https://github.com/Lewuathe/docker-hadoop-cluster <
> https://github.com/Lewuathe/docker-hadoop-cluster>
>
> And I also updated to build the latest trunk image automatically and
> upload onto Docker Hub.
> So you can easily check and test the latest trunk branch in the
> environment which is more close to actual usage.
>
> If you already installed docker and docker-compose, what needed is
> docker-compose.yml like this.
>
> version: '2'
>
> services:
>   master:
>     image: lewuathe/hadoop-master
>     ports:
>       - "9870:9870"
>       - "8088:8088"
>       - "19888:19888"
>       - "8188:8188"
>     container_name: "master"
>   slave1:
>     image: lewuathe/hadoop-slave
>     container_name: "slave1"
>     depends_on:
>       - master
>     ports:
>       - "9901:9864"
>       - "8041:8042"
>   slave2:
>     image: lewuathe/hadoop-slave
>     container_name: "slave2"
>     depends_on:
>       - master
>     ports:
>       - "9902:9864"
>       - "8042:8042"
>
> The usage in detail is described in the repository.
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md <
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md>
>
> I would be glad if you use this tool for developing and debugging and make
> our development more efficient.
> Please give me any feedbacks to me. Thanks you!
>
> ————————
> Kai Sasaki
> mail: lewuathe@me.com <ma...@me.com>
> github: https://github.com/Lewuathe <https://github.com/Lewuathe>
>
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: Multiple node Hadoop cluster on Docker for test and debugging

Posted by Zhe Zhang <zh...@gmail.com>.
Nice work Kai! Quick comment: have you considered Bigtop? It allows
creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
a custom built Hadoop (from the instructions I read, it builds from a
certain git branch; I'm not sure if that customizable).

On Wed, Nov 9, 2016 at 5:39 AM Sasaki Kai <le...@me.com> wrote:

> Hi Hadoop developers
>
> The other day I created a tool for launching multiple node hadoop cluster
> on docker container.
> You can easily launch multiple node hadoop cluster from your Hadoop source
> code.
> It is useful for testing and debugging. Actually I often use it before
> submitting a patch to Hadoop project.
> https://github.com/Lewuathe/docker-hadoop-cluster <
> https://github.com/Lewuathe/docker-hadoop-cluster>
>
> And I also updated to build the latest trunk image automatically and
> upload onto Docker Hub.
> So you can easily check and test the latest trunk branch in the
> environment which is more close to actual usage.
>
> If you already installed docker and docker-compose, what needed is
> docker-compose.yml like this.
>
> version: '2'
>
> services:
>   master:
>     image: lewuathe/hadoop-master
>     ports:
>       - "9870:9870"
>       - "8088:8088"
>       - "19888:19888"
>       - "8188:8188"
>     container_name: "master"
>   slave1:
>     image: lewuathe/hadoop-slave
>     container_name: "slave1"
>     depends_on:
>       - master
>     ports:
>       - "9901:9864"
>       - "8041:8042"
>   slave2:
>     image: lewuathe/hadoop-slave
>     container_name: "slave2"
>     depends_on:
>       - master
>     ports:
>       - "9902:9864"
>       - "8042:8042"
>
> The usage in detail is described in the repository.
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md <
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md>
>
> I would be glad if you use this tool for developing and debugging and make
> our development more efficient.
> Please give me any feedbacks to me. Thanks you!
>
> ————————
> Kai Sasaki
> mail: lewuathe@me.com <ma...@me.com>
> github: https://github.com/Lewuathe <https://github.com/Lewuathe>
>
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: Multiple node Hadoop cluster on Docker for test and debugging

Posted by Mingliang Liu <li...@apache.org>.
I know a similar tool; it has Ambari and Spark integration as well.

https://github.com/weiqingy/caochong <https://github.com/weiqingy/caochong>

Thanks,

L

> On Nov 9, 2016, at 5:38 AM, Sasaki Kai <le...@me.com> wrote:
> 
> Hi Hadoop developers
> 
> The other day I created a tool for launching multiple node hadoop cluster on docker container.
> You can easily launch multiple node hadoop cluster from your Hadoop source code. 
> It is useful for testing and debugging. Actually I often use it before submitting a patch to Hadoop project.
> https://github.com/Lewuathe/docker-hadoop-cluster <https://github.com/Lewuathe/docker-hadoop-cluster>
> 
> And I also updated to build the latest trunk image automatically and upload onto Docker Hub.
> So you can easily check and test the latest trunk branch in the environment which is more close to actual usage.
> 
> If you already installed docker and docker-compose, what needed is docker-compose.yml like this.
> 
> version: '2'
> 
> services:
>  master:
>    image: lewuathe/hadoop-master
>    ports:
>      - "9870:9870"
>      - "8088:8088"
>      - "19888:19888"
>      - "8188:8188"
>    container_name: "master"
>  slave1:
>    image: lewuathe/hadoop-slave
>    container_name: "slave1"
>    depends_on:
>      - master
>    ports:
>      - "9901:9864"
>      - "8041:8042"
>  slave2:
>    image: lewuathe/hadoop-slave
>    container_name: "slave2"
>    depends_on:
>      - master
>    ports:
>      - "9902:9864"
>      - "8042:8042"
> 
> The usage in detail is described in the repository.
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md <https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md>
> 
> I would be glad if you use this tool for developing and debugging and make our development more efficient.
> Please give me any feedbacks to me. Thanks you!
> 
> ————————
> Kai Sasaki
> mail: lewuathe@me.com <ma...@me.com>
> github: https://github.com/Lewuathe <https://github.com/Lewuathe>
> 
>