Flink Slot request bulk is not fulfillable!

[Li Shao <ls...@gmail.com>]

Hi All,
We are using flink batch mode to process s3 files. However, recently we are
seeing the errors like:


Caused by: java.util.concurrent.CompletionException:
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
Slot request bulk is not fulfillable! Could not allocate the required slot
within slot request timeout

    at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown
Source)

    at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown
Source)

    at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown
Source)

    ... 36 more

Caused by:
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
Slot request bulk is not fulfillable! Could not allocate the required slot
within slot request timeout

    at
org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:86)

    ... 29 more

Caused by: java.util.concurrent.TimeoutException: Timeout has occurred:
300000 ms

    ... 30 more


I can see the job manager & task manager still got memory to run jobs.
Anyone can help me on this?

Job manager: JVM: 5.02 GB / 6.00 GB
jobmanager.heap.size 6144m
jobmanager.memory.heap.size 6442450944b
jobmanager.memory.jvm-metaspace.size 4294967296b
jobmanager.memory.jvm-overhead.max 1073741824b
jobmanager.memory.jvm-overhead.min 1073741824b
jobmanager.memory.off-heap.size 134217728b

Re: Flink Slot request bulk is not fulfillable!

[Saurabh Singh <sa...@gmail.com>]

Hi Li,

The error suggests that Job is not able to acquire the required TaskManager
TaskSlots within the configured time duration of 5 minutes.
Job Runs on the TaskManagers (Worker Nodes). Helpful Link -
https://nightlies.apache.org/flink/flink-docs-master/docs/concepts/flink-architecture/#anatomy-of-a-flink-cluster
TaskSlot information is available in the Flink GUI as below.
If you see available as 0, and the job is not running, it means it is not
able to acquire the required resources (CPU/MEM).
For example - 1 taskslot = 1 compute resource which could be 1 CPU and 4GB
of RAM, this is the config that you define and you get possible Task Slots
for your node type.

[image: image.png]
Thanks
Saurabh

On Mon, Jul 22, 2024 at 11:01 AM Li Shao <ls...@gmail.com> wrote:

> Hi All,
> We are using flink batch mode to process s3 files. However, recently we
> are seeing the errors like:
>
>
> Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Slot request bulk is not fulfillable! Could not allocate the required slot
> within slot request timeout
>
>     at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown
> Source)
>
>     at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown
> Source)
>
>     at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown
> Source)
>
>     ... 36 more
>
> Caused by:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Slot request bulk is not fulfillable! Could not allocate the required slot
> within slot request timeout
>
>     at
> org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:86)
>
>     ... 29 more
>
> Caused by: java.util.concurrent.TimeoutException: Timeout has occurred:
> 300000 ms
>
>     ... 30 more
>
>
> I can see the job manager & task manager still got memory to run jobs.
> Anyone can help me on this?
>
> Job manager: JVM: 5.02 GB / 6.00 GB
> jobmanager.heap.size 6144m
> jobmanager.memory.heap.size 6442450944b
> jobmanager.memory.jvm-metaspace.size 4294967296b
> jobmanager.memory.jvm-overhead.max 1073741824b
> jobmanager.memory.jvm-overhead.min 1073741824b
> jobmanager.memory.off-heap.size 134217728b
>