Flink Docker 1.5.x local mode

[Thomas Weise <th...@apache.org>]

I tried to run the 1.5.0 and 1.5.1 docker containers with local mode,
following the instructions at
https://docs.docker.com/samples/library/flink/#run-a-flink-local-cluster

docker run --name flink_local -p 8081:8081 -t flink local


Starts the cluster, but it does not start a task manager. That's a change
from 1.4.2, where a TM will be started and also listed in the web UI.
Subsequent job submission will fail since no slots can be allocated. Is
this a bug or is the documentation not up to date?

Thanks,
Thomas

Re: Flink Docker 1.5.x local mode

[Thomas Weise <th...@apache.org>]

Hi Till,

Thanks, this explains the confusion.

I think there is still a benefit to run everything in a single Docker
container for development purposes. I found the next closest match to be
`./bin/start-cluster.sh`, even though that does not provide the combined
logs in the console.

I added https://github.com/docker-flink/docker-flink/issues/44

Thanks



On Mon, Jul 23, 2018 at 12:05 AM Till Rohrmann <tr...@apache.org> wrote:

> Hi Thomas,
>
> I think the docker image has not been updated wrt the changes of Flip-6. In
> Flip-6 we no longer support a `local` execution mode which starts a
> JobMaster and a TaskExecutor in the same process. I think we need to update
> the images to start both processes if one wants to have single image
> cluster. For the meantime I would recommend starting the `JobMaster` and
> `TaskExecutors` explicitly as stated here [1].
>
> [1]
>
> https://docs.docker.com/samples/library/flink/#running-a-jobmanager-or-a-taskmanager
>
> Cheers,
> Till
>
> On Mon, Jul 23, 2018 at 4:26 AM vino yang <ya...@gmail.com> wrote:
>
> > Hi Thomas,
> >
> > I watched the code of docker flink v1.5[1], it just starts job manager
> > component when you run with local mode.
> >
> > More information you can refer:
> > Flink official documentation of docker deployment :
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/docker.html
> > (out of date);
> > Docker documentation of flink :
> > https://docs.docker.com/samples/library/flink/
> >
> > [1]:
> >
> >
> https://github.com/docker-flink/docker-flink/blob/27cd2b49486a31fa449d8551be58e23c576bf44c/1.5/scala_2.11-debian/docker-entrypoint.sh#L56
> >
> > Thanks, vino.
> >
> >
> > 2018-07-23 5:58 GMT+08:00 Thomas Weise <th...@apache.org>:
> >
> > > I tried to run the 1.5.0 and 1.5.1 docker containers with local mode,
> > > following the instructions at
> > >
> https://docs.docker.com/samples/library/flink/#run-a-flink-local-cluster
> > >
> > > docker run --name flink_local -p 8081:8081 -t flink local
> > >
> > >
> > > Starts the cluster, but it does not start a task manager. That's a
> change
> > > from 1.4.2, where a TM will be started and also listed in the web UI.
> > > Subsequent job submission will fail since no slots can be allocated. Is
> > > this a bug or is the documentation not up to date?
> > >
> > > Thanks,
> > > Thomas
> > >
> >
>

Re: Flink Docker 1.5.x local mode

[Till Rohrmann <tr...@apache.org>]

Hi Thomas,

I think the docker image has not been updated wrt the changes of Flip-6. In
Flip-6 we no longer support a `local` execution mode which starts a
JobMaster and a TaskExecutor in the same process. I think we need to update
the images to start both processes if one wants to have single image
cluster. For the meantime I would recommend starting the `JobMaster` and
`TaskExecutors` explicitly as stated here [1].

[1]
https://docs.docker.com/samples/library/flink/#running-a-jobmanager-or-a-taskmanager

Cheers,
Till

On Mon, Jul 23, 2018 at 4:26 AM vino yang <ya...@gmail.com> wrote:

> Hi Thomas,
>
> I watched the code of docker flink v1.5[1], it just starts job manager
> component when you run with local mode.
>
> More information you can refer:
> Flink official documentation of docker deployment :
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/docker.html
> (out of date);
> Docker documentation of flink :
> https://docs.docker.com/samples/library/flink/
>
> [1]:
>
> https://github.com/docker-flink/docker-flink/blob/27cd2b49486a31fa449d8551be58e23c576bf44c/1.5/scala_2.11-debian/docker-entrypoint.sh#L56
>
> Thanks, vino.
>
>
> 2018-07-23 5:58 GMT+08:00 Thomas Weise <th...@apache.org>:
>
> > I tried to run the 1.5.0 and 1.5.1 docker containers with local mode,
> > following the instructions at
> > https://docs.docker.com/samples/library/flink/#run-a-flink-local-cluster
> >
> > docker run --name flink_local -p 8081:8081 -t flink local
> >
> >
> > Starts the cluster, but it does not start a task manager. That's a change
> > from 1.4.2, where a TM will be started and also listed in the web UI.
> > Subsequent job submission will fail since no slots can be allocated. Is
> > this a bug or is the documentation not up to date?
> >
> > Thanks,
> > Thomas
> >
>

Re: Flink Docker 1.5.x local mode

[vino yang <ya...@gmail.com>]

Hi Thomas,

I watched the code of docker flink v1.5[1], it just starts job manager
component when you run with local mode.

More information you can refer:
Flink official documentation of docker deployment :
https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/docker.html
(out of date);
Docker documentation of flink :
https://docs.docker.com/samples/library/flink/

[1]:
https://github.com/docker-flink/docker-flink/blob/27cd2b49486a31fa449d8551be58e23c576bf44c/1.5/scala_2.11-debian/docker-entrypoint.sh#L56

Thanks, vino.


2018-07-23 5:58 GMT+08:00 Thomas Weise <th...@apache.org>:

> I tried to run the 1.5.0 and 1.5.1 docker containers with local mode,
> following the instructions at
> https://docs.docker.com/samples/library/flink/#run-a-flink-local-cluster
>
> docker run --name flink_local -p 8081:8081 -t flink local
>
>
> Starts the cluster, but it does not start a task manager. That's a change
> from 1.4.2, where a TM will be started and also listed in the web UI.
> Subsequent job submission will fail since no slots can be allocated. Is
> this a bug or is the documentation not up to date?
>
> Thanks,
> Thomas
>