You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Alexander Semeshchenko <as...@gmail.com> on 2020/10/07 21:03:29 UTC
NoResourceAvailableException
<https://stackoverflow.com/posts/64252040/timeline>
Installing (download & tar zxf) Apache Flink 1.11.1 and running: ./bin/flink
run examples/streaming/WordCount.jar it show on the nice message after more
less 5 min. the trying of submitting: Caused by:
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
Could not allocate the required slot within slot request timeout. Please
make sure that the cluster has enough resources. at
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441)
... 45 more Caused by: java.util.concurrent.CompletionException:
java.util.concurrent.TimeoutException at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
It's Flink default configuration.
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little
Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per
socket: 1
free -g total used free shared buff/cache available
Mem: 62 1 23 3 37 57 Swap: 7 0 7
are there some advices about what is happened?
Re: NoResourceAvailableException
Posted by Khachatryan Roman <kh...@gmail.com>.
Hi Alexander,
Thanks for sharing,
I see a lot of exceptions in the logs, particularly
*Caused by: java.net.BindException: Could not start actor system on any
port in port range 6123
which means that there's probably more than one instance running and is
likely the root cause.
So it makes sense to make sure that the previous attempts cleaned up.
Regards,
Roman
On Tue, Oct 20, 2020 at 12:08 AM Alexander Semeshchenko <as...@gmail.com>
wrote:
> Hi Roman,
> I made the cluster: 1 master, 2 worker. All - 8 cpu, 32 g RAM . Red Hat
> Enterprise Linux Server release 7.9 (Maipo)
> vsmart-f01 - master
> vsmart-f02 - worker
> vsmart-f03 - worker
> tvsmart-f02askmanager.numberOfTaskSlots for each node is : 8
>
> Then:
> *[flink@vsmart-f01 flink-1.11.1]$ ./bin/start-cluster.sh *
> *Starting cluster.*
> *[INFO] 1 instance(s) of standalonesession are already running on
> vsmart-f01.*
> *Starting standalonesession daemon on host vsmart-f01.*
> *flink@10.92.194.19 <fl...@10.92.194.19>'s password: *
> *[INFO] 1 instance(s) of taskexecutor are already running on vsmart-f02.*
> *Starting taskexecutor daemon on host vsmart-f02.*
> *flink@10.92.194.20 <fl...@10.92.194.20>'s password: *
> *Starting taskexecutor daemon on host vsmart-f03.*
>
> The cluster start up, running WordCount from master:
> *./bin/flink run -c org.apache.flink.examples.java.wordcount.WordCount
> ./examples/batch/WordCount.jar --output file:/tmp/wordcount_out*
>
> After 5 min. the job was canceled.
> In the screenshot appeared that was never assigned taskmanager for the job
> operator.
> I've put the 3 logs( from each node) here.
>
> Thanks and Best Regards.
> Alex
>
>
> On Mon, Oct 19, 2020 at 5:47 PM Khachatryan Roman <
> khachatryan.roman@gmail.com> wrote:
>
>> Hi Alex,
>>
>> This message isn't actually a problem - netty can't find the native
>> transports and falls back to nio-based one.
>> Does increasing taskmanager.numberOfTaskSlots in flink-conf.yaml help?
>> Can you share the full logs in DEBUG mode?
>>
>> Regards,
>> Roman
>>
>>
>> On Mon, Oct 19, 2020 at 6:14 PM Alexander Semeshchenko <as...@gmail.com>
>> wrote:
>>
>>> thank you for your response.
>>>
>>> taskmanager has 1 slot , 1 slot free but WordCount job never change its
>>> status from "Created".
>>> After more less 5 min. job is canceled.
>>> I attached screenshot of taskmanager.
>>>
>>> Best Regards
>>> Alexander
>>>
>>> On Wed, Oct 14, 2020 at 6:13 PM Khachatryan Roman <
>>> khachatryan.roman@gmail.com> wrote:
>>>
>>>> Hi,
>>>> Thanks for sharing the details and sorry for the late reply.
>>>> You can check the number of free slots in the task manager in the web
>>>> UI (http://localhost:8081/#/task-manager by default).
>>>> Before running the program, there should be 1 TM with 1 slot available
>>>> which should be free (with default settings).
>>>>
>>>> If there are other jobs, you can increase slots per TM by increasing
>>>> taskmanager.numberOfTaskSlots in flink-conf.yaml [1].
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#taskmanager-numberoftaskslots
>>>>
>>>> Regards,
>>>> Roman
>>>>
>>>>
>>>> On Wed, Oct 14, 2020 at 6:56 PM Alexander Semeshchenko <
>>>> as777us@gmail.com> wrote:
>>>>
>>>>> Hi, is there any news about my issue "Flink -
>>>>> NoResourceAvailableException " post - installed WordCount job ?
>>>>> Best
>>>>>
>>>>> On Fri, Oct 9, 2020 at 10:19 AM Alexander Semeshchenko <
>>>>> as777us@gmail.com> wrote:
>>>>>
>>>>>> Yes, I made the following accions:
>>>>>> - download Flink
>>>>>> - ./bin/start-cluster.sh.
>>>>>> - ./bin/flink run ./examples/streaming/WordCount.jar
>>>>>> ------------------------------------------------
>>>>>> Then, tried to increase values for > ulimit , VM memory values...
>>>>>> Below I put the logs messages.
>>>>>>
>>>>>> It's rare as I could do the same job on: My Macbook( 8 cpu, 16g RAM
>>>>>> ), on k8s cluster - 4 cpu, 8g RAM
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 9, 2020 at 3:32 AM Khachatryan Roman <
>>>>>> khachatryan.roman@gmail.com> wrote:
>>>>>>
>>>>>>> I assume that before submitting a job you started a cluster with
>>>>>>> default settings with ./bin/start-cluster.sh.
>>>>>>>
>>>>>>> Did you submit any other jobs?
>>>>>>> Can you share the logs from log folder?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Roman
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Oct 7, 2020 at 11:03 PM Alexander Semeshchenko <
>>>>>>> as777us@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> <https://stackoverflow.com/posts/64252040/timeline>
>>>>>>>>
>>>>>>>> Installing (download & tar zxf) Apache Flink 1.11.1 and running: ./bin/flink
>>>>>>>> run examples/streaming/WordCount.jar it show on the nice message
>>>>>>>> after more less 5 min. the trying of submitting: Caused by:
>>>>>>>> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
>>>>>>>> Could not allocate the required slot within slot request timeout. Please
>>>>>>>> make sure that the cluster has enough resources. at
>>>>>>>> org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441)
>>>>>>>> ... 45 more Caused by: java.util.concurrent.CompletionException:
>>>>>>>> java.util.concurrent.TimeoutException at
>>>>>>>> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
>>>>>>>> at
>>>>>>>> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
>>>>>>>> at
>>>>>>>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
>>>>>>>> at
>>>>>>>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
>>>>>>>>
>>>>>>>> It's Flink default configuration.
>>>>>>>>
>>>>>>>> Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order:
>>>>>>>> Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1
>>>>>>>> Core(s) per socket: 1
>>>>>>>> free -g total used free shared buff/cache available
>>>>>>>>
>>>>>>>> Mem: 62 1 23 3 37 57 Swap: 7 0 7
>>>>>>>>
>>>>>>>> are there some advices about what is happened?
>>>>>>>>
>>>>>>>
Re: NoResourceAvailableException
Posted by Khachatryan Roman <kh...@gmail.com>.
Hi Alex,
This message isn't actually a problem - netty can't find the native
transports and falls back to nio-based one.
Does increasing taskmanager.numberOfTaskSlots in flink-conf.yaml help?
Can you share the full logs in DEBUG mode?
Regards,
Roman
On Mon, Oct 19, 2020 at 6:14 PM Alexander Semeshchenko <as...@gmail.com>
wrote:
> thank you for your response.
>
> taskmanager has 1 slot , 1 slot free but WordCount job never change its
> status from "Created".
> After more less 5 min. job is canceled.
> I attached screenshot of taskmanager.
>
> Best Regards
> Alexander
>
> On Wed, Oct 14, 2020 at 6:13 PM Khachatryan Roman <
> khachatryan.roman@gmail.com> wrote:
>
>> Hi,
>> Thanks for sharing the details and sorry for the late reply.
>> You can check the number of free slots in the task manager in the web UI (
>> http://localhost:8081/#/task-manager by default).
>> Before running the program, there should be 1 TM with 1 slot available
>> which should be free (with default settings).
>>
>> If there are other jobs, you can increase slots per TM by increasing
>> taskmanager.numberOfTaskSlots in flink-conf.yaml [1].
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#taskmanager-numberoftaskslots
>>
>> Regards,
>> Roman
>>
>>
>> On Wed, Oct 14, 2020 at 6:56 PM Alexander Semeshchenko <as...@gmail.com>
>> wrote:
>>
>>> Hi, is there any news about my issue "Flink -
>>> NoResourceAvailableException " post - installed WordCount job ?
>>> Best
>>>
>>> On Fri, Oct 9, 2020 at 10:19 AM Alexander Semeshchenko <
>>> as777us@gmail.com> wrote:
>>>
>>>> Yes, I made the following accions:
>>>> - download Flink
>>>> - ./bin/start-cluster.sh.
>>>> - ./bin/flink run ./examples/streaming/WordCount.jar
>>>> ------------------------------------------------
>>>> Then, tried to increase values for > ulimit , VM memory values...
>>>> Below I put the logs messages.
>>>>
>>>> It's rare as I could do the same job on: My Macbook( 8 cpu, 16g RAM ),
>>>> on k8s cluster - 4 cpu, 8g RAM
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>> On Fri, Oct 9, 2020 at 3:32 AM Khachatryan Roman <
>>>> khachatryan.roman@gmail.com> wrote:
>>>>
>>>>> I assume that before submitting a job you started a cluster with
>>>>> default settings with ./bin/start-cluster.sh.
>>>>>
>>>>> Did you submit any other jobs?
>>>>> Can you share the logs from log folder?
>>>>>
>>>>> Regards,
>>>>> Roman
>>>>>
>>>>>
>>>>> On Wed, Oct 7, 2020 at 11:03 PM Alexander Semeshchenko <
>>>>> as777us@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> <https://stackoverflow.com/posts/64252040/timeline>
>>>>>>
>>>>>> Installing (download & tar zxf) Apache Flink 1.11.1 and running: ./bin/flink
>>>>>> run examples/streaming/WordCount.jar it show on the nice message
>>>>>> after more less 5 min. the trying of submitting: Caused by:
>>>>>> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
>>>>>> Could not allocate the required slot within slot request timeout. Please
>>>>>> make sure that the cluster has enough resources. at
>>>>>> org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441)
>>>>>> ... 45 more Caused by: java.util.concurrent.CompletionException:
>>>>>> java.util.concurrent.TimeoutException at
>>>>>> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
>>>>>> at
>>>>>> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
>>>>>> at
>>>>>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
>>>>>> at
>>>>>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
>>>>>>
>>>>>> It's Flink default configuration.
>>>>>>
>>>>>> Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order:
>>>>>> Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1
>>>>>> Core(s) per socket: 1
>>>>>> free -g total used free shared buff/cache available
>>>>>>
>>>>>> Mem: 62 1 23 3 37 57 Swap: 7 0 7
>>>>>>
>>>>>> are there some advices about what is happened?
>>>>>>
>>>>>
Re: NoResourceAvailableException
Posted by Khachatryan Roman <kh...@gmail.com>.
I assume that before submitting a job you started a cluster with default
settings with ./bin/start-cluster.sh.
Did you submit any other jobs?
Can you share the logs from log folder?
Regards,
Roman
On Wed, Oct 7, 2020 at 11:03 PM Alexander Semeshchenko <as...@gmail.com>
wrote:
>
> <https://stackoverflow.com/posts/64252040/timeline>
>
> Installing (download & tar zxf) Apache Flink 1.11.1 and running: ./bin/flink
> run examples/streaming/WordCount.jar it show on the nice message after
> more less 5 min. the trying of submitting: Caused by:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Could not allocate the required slot within slot request timeout. Please
> make sure that the cluster has enough resources. at
> org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441)
> ... 45 more Caused by: java.util.concurrent.CompletionException:
> java.util.concurrent.TimeoutException at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
>
> It's Flink default configuration.
>
> Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little
> Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per
> socket: 1
> free -g total used free shared buff/cache available
>
> Mem: 62 1 23 3 37 57 Swap: 7 0 7
>
> are there some advices about what is happened?
>