Setting Parallelsim for implicit key extractor operator/NoResourceAvailableException: Not enough free slots available to run the job.

["Chan, Regina" <Re...@gs.com>]

Hi,

I'm currently running multiple jobs in the same yarn session and controlling how I submit jobs based off what my calculated parallelism should be. For example, I have 4 TMs, 2 slots, so I should be able run multiple parallelism 1 jobs at the same time up to 8. However, what I'm seeing is that the implicit key extractor that's going into the cogroup has parallelism 8 instead of having the same parallelism of the cogroup 1. This causes the other jobs to fail with:

org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Not enough free slots available to run the job. You can decrease the operator parallelism or increase the number of slots per TaskManager in the configuration.
This isn't an issue when it chains but is one when the key extractor becomes its own operator. Is there a way I can force the key extractor to have the correct parallelism? What's the best way around this?






[cid:image001.png@01D3E0AF.95D96F70]


Regina Chan
Goldman Sachs - Enterprise Platforms, Data Architecture
30 Hudson Street, 37th floor | Jersey City, NY 07302 *  (212) 902-5697

Re: Setting Parallelsim for implicit key extractor operator/NoResourceAvailableException: Not enough free slots available to run the job.

[Fabian Hueske <fh...@gmail.com>]

Hi Regina,

Thanks for reporting this issue!
This is a bug and should be fixed. I've created a JIRA issue [1].

As a workaround, you can set the default parallelism of the
ExecutionEnvironment:

env.setParallelism(2);

The generated operator will then have the configured default parallelism.

Best,
Fabian

[1] https://issues.apache.org/jira/browse/FLINK-9289

2018-05-01 0:27 GMT+02:00 Chan, Regina <Re...@gs.com>:

> Hi,
>
>
>
> I’m currently running multiple jobs in the same yarn session and
> controlling how I submit jobs based off what my calculated parallelism
> should be. For example, I have 4 TMs, 2 slots, so I should be able run
> multiple parallelism 1 jobs at the same time up to 8. However, what I’m
> seeing is that the implicit key extractor that’s going into the cogroup has
> parallelism 8 instead of having the same parallelism of the cogroup 1. This
> causes the other jobs to fail with:
>
>
>
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Not enough free slots available to run the job. You can decrease the
> operator parallelism or increase the number of slots per TaskManager in the
> configuration.
>
> This isn’t an issue when it chains but is one when the key extractor
> becomes its own operator. Is there a way I can force the key extractor to
> have the correct parallelism? What’s the best way around this?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Regina Chan*
>
> *Goldman Sachs* *–* Enterprise Platforms, Data Architecture
>
> *30 Hudson Street, 37th floor | Jersey City, NY 07302
> <https://maps.google.com/?q=30+Hudson+Street,+37th+floor+%7C+Jersey+City,+NY+07302&entry=gmail&source=g>*
> (  (212) 902-5697
>
>
>