You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by shyla deshpande <de...@gmail.com> on 2018/08/15 05:23:35 UTC
docker, error NoResourceAvailableException..
Hello all,
Trying to use docker as a single node flink cluster.
docker run --name flink_local -p 8081:8081 -t flink local
I submited a job to the cluster using the Web UI. The job failed. I see
this error message in the docker logs.
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
Could not allocate all requires slots within timeout of 300000 ms. Slots
required: 2, slots allocated: 0
The Web UI, shows 0 taskmanagers and 0 task slots on the Flink dashboard.
How do I start the docker with 2 Task slots?
Appreciate any help.
Thanks
Re: docker, error NoResourceAvailableException..
Posted by Esteban Serrano <st...@gmail.com>.
You can also instead of defining 2 services (taskmanager and taskmanager1),
set the scale parameter on taskmanager to the number of desired slots.
Something like this:
taskmanager:
image: "${FLINK_DOCKER_IMAGE:-flink:1.5.2}"
scale: 2
expose:
- "6121"
- "6122"
- "8081"
On Wed, Aug 15, 2018 at 1:53 PM shyla deshpande <de...@gmail.com>
wrote:
> Thanks Dominik, I will try that.
>
> On Wed, Aug 15, 2018 at 3:10 AM, Dominik Wosiński <wo...@gmail.com>
> wrote:
>
>> Hey,
>> The problem is that your command does start Job Manager container, but it
>> does not start the Task Manager . That is why you have 0 slots. Currently,
>> the default *numberOfTaskSlots* is set to the number of CPUs avaialbe on
>> the machine.
>>
>>
>> So, You generally can to do 2 things:
>>
>>
>> 1) Start Job Manager and 2 Task Managers. If you have Docker Compose
>> available, you can paste this to your *docker-compose.yml* :
>>
>>
>>
>> *services: jobmanager: image: *${FLINK_DOCKER_IMAGE_NAME:-flink}
>>
>> *expose: *- "6123"
>>
>> *ports: *- "8081:8081"
>> *command: *jobmanager
>>
>> *environment: *- JOB_MANAGER_RPC_ADDRESS=jobmanager
>>
>>
>> *taskmanager: image: *${FLINK_DOCKER_IMAGE_NAME:-flink}
>>
>> *expose: *- "6121"
>> - "6122"
>>
>> *depends_on: *- jobmanager
>> *command: *taskmanager
>>
>> *links: *- "jobmanager:jobmanager"
>>
>> *environment: *- JOB_MANAGER_RPC_ADDRESS=jobmanager
>>
>>
>> *taskmanager1: image: *${FLINK_DOCKER_IMAGE_NAME:-flink}
>>
>> *expose: *- "6190"
>> - "6120"
>>
>> *depends_on: *- jobmanager
>> *command: *taskmanager
>>
>> *links: *- "jobmanager:jobmanager"
>>
>> *environment: *- JOB_MANAGER_RPC_ADDRESS=jobmanager
>>
>>
>>
>> This will give you 1 Job Manager and 2 Task Managers with one task slot
>> each, so 2 Task slots in general.
>>
>> 2) You can deploy 1 Job Manager and 1 Task Manager.Then you need to
>> modify *flink-conf.yml* and set the following setting :
>>
>> *taskmanager.numberOfTaskSlots: *2
>>
>>
>> This will give you 2 Task Slots with only 1 Task Manager. But you will
>> need to somehow override config in the container, possibly using :
>> https://docs.docker.com/storage/volumes/
>>
>> Regards,
>> Dominik.
>>
>> *Od: *shyla deshpande <de...@gmail.com>
>> *Wysłano: *środa, 15 sierpnia 2018 07:23
>> *Do: *user <us...@flink.apache.org>
>> *Temat: *docker, error NoResourceAvailableException..
>>
>>
>>
>> Hello all,
>>
>>
>>
>> Trying to use docker as a single node flink cluster.
>>
>>
>>
>> docker run --name flink_local -p 8081:8081 -t flink local
>>
>>
>>
>> I submited a job to the cluster using the Web UI. The job failed. I see
>> this error message in the docker logs.
>>
>>
>>
>> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
>> Could not allocate all requires slots within timeout of 300000 ms. Slots
>> required: 2, slots allocated: 0
>>
>>
>>
>> The Web UI, shows 0 taskmanagers and 0 task slots on the Flink dashboard.
>>
>> How do I start the docker with 2 Task slots?
>>
>>
>>
>> Appreciate any help.
>>
>>
>>
>> Thanks
>>
>>
>>
>
>
Re: docker, error NoResourceAvailableException..
Posted by shyla deshpande <de...@gmail.com>.
Thanks Dominik, I will try that.
On Wed, Aug 15, 2018 at 3:10 AM, Dominik Wosiński <wo...@gmail.com> wrote:
> Hey,
> The problem is that your command does start Job Manager container, but it
> does not start the Task Manager . That is why you have 0 slots. Currently,
> the default *numberOfTaskSlots* is set to the number of CPUs avaialbe on
> the machine.
>
>
> So, You generally can to do 2 things:
>
>
> 1) Start Job Manager and 2 Task Managers. If you have Docker Compose
> available, you can paste this to your *docker-compose.yml* :
>
>
>
> *services: jobmanager: image: *${FLINK_DOCKER_IMAGE_NAME:-flink}
>
> *expose: *- "6123"
>
> *ports: *- "8081:8081"
> *command: *jobmanager
>
> *environment: *- JOB_MANAGER_RPC_ADDRESS=jobmanager
>
>
> *taskmanager: image: *${FLINK_DOCKER_IMAGE_NAME:-flink}
>
> *expose: *- "6121"
> - "6122"
>
> *depends_on: *- jobmanager
> *command: *taskmanager
>
> *links: *- "jobmanager:jobmanager"
>
> *environment: *- JOB_MANAGER_RPC_ADDRESS=jobmanager
>
>
> *taskmanager1: image: *${FLINK_DOCKER_IMAGE_NAME:-flink}
>
> *expose: *- "6190"
> - "6120"
>
> *depends_on: *- jobmanager
> *command: *taskmanager
>
> *links: *- "jobmanager:jobmanager"
>
> *environment: *- JOB_MANAGER_RPC_ADDRESS=jobmanager
>
>
>
> This will give you 1 Job Manager and 2 Task Managers with one task slot
> each, so 2 Task slots in general.
>
> 2) You can deploy 1 Job Manager and 1 Task Manager.Then you need to modify
> *flink-conf.yml* and set the following setting :
>
> *taskmanager.numberOfTaskSlots: *2
>
>
> This will give you 2 Task Slots with only 1 Task Manager. But you will
> need to somehow override config in the container, possibly using :
> https://docs.docker.com/storage/volumes/
>
> Regards,
> Dominik.
>
> *Od: *shyla deshpande <de...@gmail.com>
> *Wysłano: *środa, 15 sierpnia 2018 07:23
> *Do: *user <us...@flink.apache.org>
> *Temat: *docker, error NoResourceAvailableException..
>
>
>
> Hello all,
>
>
>
> Trying to use docker as a single node flink cluster.
>
>
>
> docker run --name flink_local -p 8081:8081 -t flink local
>
>
>
> I submited a job to the cluster using the Web UI. The job failed. I see
> this error message in the docker logs.
>
>
>
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Could not allocate all requires slots within timeout of 300000 ms. Slots
> required: 2, slots allocated: 0
>
>
>
> The Web UI, shows 0 taskmanagers and 0 task slots on the Flink dashboard.
>
> How do I start the docker with 2 Task slots?
>
>
>
> Appreciate any help.
>
>
>
> Thanks
>
>
>
ODP: docker, error NoResourceAvailableException..
Posted by Dominik Wosiński <wo...@gmail.com>.
Hey,
The problem is that your command does start Job Manager container, but it does not start the Task Manager . That is why you have 0 slots. Currently, the default numberOfTaskSlots is set to the number of CPUs avaialbe on the machine.
So, You generally can to do 2 things:
1) Start Job Manager and 2 Task Managers. If you have Docker Compose available, you can paste this to your docker-compose.yml :
services:
jobmanager:
image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
expose:
- "6123"
ports:
- "8081:8081"
command: jobmanager
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
taskmanager:
image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
expose:
- "6121"
- "6122"
depends_on:
- jobmanager
command: taskmanager
links:
- "jobmanager:jobmanager"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
taskmanager1:
image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
expose:
- "6190"
- "6120"
depends_on:
- jobmanager
command: taskmanager
links:
- "jobmanager:jobmanager"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
This will give you 1 Job Manager and 2 Task Managers with one task slot each, so 2 Task slots in general.
2) You can deploy 1 Job Manager and 1 Task Manager.Then you need to modify flink-conf.yml and set the following setting :
taskmanager.numberOfTaskSlots: 2
This will give you 2 Task Slots with only 1 Task Manager. But you will need to somehow override config in the container, possibly using : https://docs.docker.com/storage/volumes/
Regards,
Dominik.
Od: shyla deshpande
Wysłano: środa, 15 sierpnia 2018 07:23
Do: user
Temat: docker, error NoResourceAvailableException..
Hello all,
Trying to use docker as a single node flink cluster.
docker run --name flink_local -p 8081:8081 -t flink local
I submited a job to the cluster using the Web UI. The job failed. I see this error message in the docker logs.
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate all requires slots within timeout of 300000 ms. Slots required: 2, slots allocated: 0
The Web UI, shows 0 taskmanagers and 0 task slots on the Flink dashboard.
How do I start the docker with 2 Task slots?
Appreciate any help.
Thanks