You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Nikos Spanos <sp...@outlook.com> on 2023/09/22 06:38:16 UTC
Deploy multi-node Hadoop with Docker
Hi,
I am creating a multi-node Hadoop cluster for a personal project, and I would like to use the official docker image (apache/hadoop<https://hub.docker.com/r/apache/hadoop>).
However, looking at the official docker image documentation and the docker-compose file I have seen the following environment variable:
environment:
ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
I would like to know if it is safe to create the namenode directory in the /tmp folder since this kind of folder is neither secure nor data persistent. Thus, I would like to understand which path is the best practice for this. Moreover, which are other environment variables I could use of.
Thanks a lot, in advance.
Kind regards,
Nikos Spanos
M.Sc Business Analytics & Big Data| Athens University of Economics & Business
Phone Number: +306982310494
Linkedin profile <http://linkedin.com/in/spanosnikos>
Re: Deploy multi-node Hadoop with Docker
Posted by Wei-Chiu Chuang <we...@apache.org>.
The Hadoop's docker image is not for production use. That's why
But we should update that if people are thinking to use it for production.
Not familiar with docker compose but contributions welcomed:
https://github.com/apache/hadoop/blob/docker-hadoop-3/docker-compose.yaml
On Fri, Sep 22, 2023 at 5:44 AM Nikos Spanos <sp...@outlook.com>
wrote:
> Hi,
>
>
>
> I am creating a multi-node Hadoop cluster for a personal project, and I
> would like to use the official docker image (apache/hadoop
> <https://hub.docker.com/r/apache/hadoop>).
>
>
>
> However, looking at the official docker image documentation and the
> docker-compose file I have seen the following environment variable:
>
>
>
> environment:
>
> ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
>
>
>
> I would like to know if it is safe to create the namenode directory in the
> /tmp folder since this kind of folder is neither secure nor data
> persistent. Thus, I would like to understand which path is the best
> practice for this. Moreover, which are other environment variables I could
> use of.
>
>
>
> Thanks a lot, in advance.
>
>
>
> Kind regards,
>
>
>
> *Nikos Spanos*
>
>
>
> M.Sc Business Analytics & Big Data| Athens University of Economics &
> Business
>
> Phone Number: +306982310494
>
> Linkedin profile <http://linkedin.com/in/spanosnikos>
>
>
>
Re: Deploy multi-node Hadoop with Docker
Posted by Ayush Saxena <ay...@gmail.com>.
Hi Nikos,
I think you are talking about the documentation in the overview
section of the docker image: https://hub.docker.com/r/apache/hadoop
I just wrote that 2-3 Months back particularly for dev purposes not
for any prod use case, you should change those values accordingly. The
docker-compose file I copied from
https://github.com/apache/hadoop/blob/docker-hadoop-3/docker-compose.yaml
-Ayush
On Fri, 22 Sept 2023 at 22:28, Nikos Spanos <sp...@outlook.com> wrote:
>
> Hi,
>
>
>
> I am creating a multi-node Hadoop cluster for a personal project, and I would like to use the official docker image (apache/hadoop).
>
>
>
> However, looking at the official docker image documentation and the docker-compose file I have seen the following environment variable:
>
>
>
> environment:
>
> ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
>
>
>
> I would like to know if it is safe to create the namenode directory in the /tmp folder since this kind of folder is neither secure nor data persistent. Thus, I would like to understand which path is the best practice for this. Moreover, which are other environment variables I could use of.
>
>
>
> Thanks a lot, in advance.
>
>
>
> Kind regards,
>
>
>
> Nikos Spanos
>
>
>
> M.Sc Business Analytics & Big Data| Athens University of Economics & Business
>
> Phone Number: +306982310494
>
> Linkedin profile
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org