You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Dipak Tandel <di...@gmail.com> on 2024/03/29 08:04:59 UTC

Batch Job with Adaptive Batch Scheduler failing with JobInitializationException: Could not start the JobMaster

Hi Everyone

I am facing some issues while running the batch job on a Flink cluster
using Adaptive Batch Scheduler. I have deployed a flink cluster on
Kubernetes using the flink Kubernetes operator and submitted a job to the
cluster using Apache beam FlinkRunner.  I am using Flink version 1.16.   I
want to figure out two things.



*1.* I am trying to run one batch job with an adaptive batch scheduler.  I
referred to the elastic scaling document [1] and based on the document I
have added the below config to my FlinkDeployment. But the job is failing
with JobInitializationException: Could not start the JobMaster. I have
attached the job manager log below. *Can someone explain why the job is
failing despite using the recommended configuration?*

jobmanager.scheduler: AdaptiveBatch
parallelism.default: "-1"
taskmanager.network.memory.buffers-per-channel: "0"
execution.batch-shuffle-mode: "ALL_EXCHANGES_BLOCKING"



*2. *The document has listed some limitations of elastic scaling. One of
them is about the input file format. *My code is written in Apache Beam,
how to check, how the input is being read in Flink?*


   - FileInputFormat sources are not supported: FileInputFormat sources are
   not supported, including StreamExecutionEnvironment#readFile(...)
   StreamExecutionEnvironment#readTextFile(...) and
StreamExecutionEnvironment#createInput(FileInputFormat,
   ...). Users should use the new sources(FileSystem DataStream Connector
   <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/filesystem/>
    or FileSystem SQL Connector
   <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/>)
   to read files when using the Adaptive Batch Scheduler.


Let me know if additional information is needed to debug the issue.

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/#adaptive-batch-scheduler

Regards
Dipak

Re: Batch Job with Adaptive Batch Scheduler failing with JobInitializationException: Could not start the JobMaster

Posted by Dipak Tandel <di...@gmail.com>.
Hi Junrui, Thank you for bringing this to my attention. I appreciate your
help in fixing the issue. I have gone through the latest Flink version 1.19
documentation
https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/elastic_scaling/#limitations-2
and I am trying to set execution mode to BATCH_FORCED but could not figure
out how to do it. I could not find any option to override the execution
mode in the configuration document
https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/,
so I added execution.mode: BATCH_FORCED config to my FlinkDeployment but
it's not working. it is still using the default value PIPELINED, I
validated it using the WebUI. Can you please help me figure out how to
override the default value? What should be the value of the key to override
the execution mode? [image: image.png]

[image: image.png]
Once again thank you for your help. Best regards, Dipak

On Fri, Mar 29, 2024 at 2:31 PM Junrui Lee <jr...@gmail.com> wrote:

> Hi Dipak,
>
> Regarding question 1, I noticed from the logs that the method
> createBatchExecutionEnvironment from Beam is being used in your job. IIUC,
> this method utilizes Flink's DataSet API. If indeed the DataSet API is
> being used, the configuration option execution.batch-shuffle-mode will not
> take effect, and you should set the ExecutionMode to BATCH_FORCED, in Beam
> you can configure executionModeForBatch to BATCH_FORCED.
> And this limitation has been documented in the latest Flink version 1.19
> documentation:
> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/elastic_scaling/#limitations-2
> .
>
> Best,
> Junrui
>
> Dipak Tandel <di...@gmail.com> 于2024年3月29日周五 16:05写道:
>
>> Hi Everyone
>>
>> I am facing some issues while running the batch job on a Flink cluster
>> using Adaptive Batch Scheduler. I have deployed a flink cluster on
>> Kubernetes using the flink Kubernetes operator and submitted a job to the
>> cluster using Apache beam FlinkRunner.  I am using Flink version 1.16.   I
>> want to figure out two things.
>>
>>
>>
>> *1.* I am trying to run one batch job with an adaptive batch scheduler.
>> I referred to the elastic scaling document [1] and based on the document I
>> have added the below config to my FlinkDeployment. But the job is failing
>> with JobInitializationException: Could not start the JobMaster. I have
>> attached the job manager log below. *Can someone explain why the job is
>> failing despite using the recommended configuration?*
>>
>> jobmanager.scheduler: AdaptiveBatch
>> parallelism.default: "-1"
>> taskmanager.network.memory.buffers-per-channel: "0"
>> execution.batch-shuffle-mode: "ALL_EXCHANGES_BLOCKING"
>>
>>
>>
>> *2. *The document has listed some limitations of elastic scaling. One of
>> them is about the input file format. *My code is written in Apache Beam,
>> how to check, how the input is being read in Flink?*
>>
>>
>>    - FileInputFormat sources are not supported: FileInputFormat sources
>>    are not supported, including StreamExecutionEnvironment#readFile(...)
>>    StreamExecutionEnvironment#readTextFile(...) and StreamExecutionEnvironment#createInput(FileInputFormat,
>>    ...). Users should use the new sources(FileSystem DataStream Connector
>>    <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/filesystem/>
>>     or FileSystem SQL Connector
>>    <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/>)
>>    to read files when using the Adaptive Batch Scheduler.
>>
>>
>> Let me know if additional information is needed to debug the issue.
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/#adaptive-batch-scheduler
>>
>> Regards
>> Dipak
>>
>>

Re: Batch Job with Adaptive Batch Scheduler failing with JobInitializationException: Could not start the JobMaster

Posted by Dipak Tandel <di...@gmail.com>.
Hi Junrui

Thanks for the reply. I added the config you mentioned above and I was able
to run a pipeline on a small data.

The same pipeline doesn't work on large data. I will create another email
thread for that issue.

Regards
Dipak

On Fri, Mar 29, 2024 at 5:24 PM Junrui Lee <jr...@gmail.com> wrote:

> Hi, Dipak
>
> To set the execution mode to BATCH_FORCED in Flink, it's not configurable
> via config option. Instead, use the Flink API with:
>
> environment.getExecutionConfig().setExecutionMode(ExecutionMode.BATCH_FORCED);
> Moreover, considering that you mentioned using Apache Beam, the
> executionModeForBatch option might help you, more details can be found
> here: https://beam.apache.org/documentation/runners/flink/.
>
> Best,
>
> Junrui
>
> Dipak Tandel <di...@gmail.com> 于2024年3月29日周五 19:21写道:
>
>> Hi Junrui,
>>
>> Thank you for bringing this to my attention. I appreciate your help in
>> fixing the issue. I have gone through the latest Flink version 1.19
>> documentation
>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/elastic_scaling/#limitations-2 and
>> I am trying to set execution mode to BATCH_FORCED but could not figure out
>> how to do it.
>>
>> I could not find any option to override the execution mode in the
>> configuration document
>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/,
>> so I added execution.mode: BATCH_FORCED config to my FlinkDeployment but
>> it's not working. it is still using the default value PIPELINED, I
>> validated it using the WebUI. Can you please help me figure out how to
>> override the default value? What should be the value of the config key to
>> override the execution mode?
>>
>>
>>   [image: image.png]
>>
>> [image: image.png]
>> Once again thank you for your help.
>>
>> Best regards,
>> Dipak
>>
>> On Fri, Mar 29, 2024 at 2:31 PM Junrui Lee <jr...@gmail.com> wrote:
>>
>>> Hi Dipak,
>>>
>>> Regarding question 1, I noticed from the logs that the method
>>> createBatchExecutionEnvironment from Beam is being used in your job. IIUC,
>>> this method utilizes Flink's DataSet API. If indeed the DataSet API is
>>> being used, the configuration option execution.batch-shuffle-mode will not
>>> take effect, and you should set the ExecutionMode to BATCH_FORCED, in Beam
>>> you can configure executionModeForBatch to BATCH_FORCED.
>>> And this limitation has been documented in the latest Flink version 1.19
>>> documentation:
>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/elastic_scaling/#limitations-2
>>> .
>>>
>>> Best,
>>> Junrui
>>>
>>> Dipak Tandel <di...@gmail.com> 于2024年3月29日周五 16:05写道:
>>>
>>>> Hi Everyone
>>>>
>>>> I am facing some issues while running the batch job on a Flink cluster
>>>> using Adaptive Batch Scheduler. I have deployed a flink cluster on
>>>> Kubernetes using the flink Kubernetes operator and submitted a job to the
>>>> cluster using Apache beam FlinkRunner.  I am using Flink version 1.16.   I
>>>> want to figure out two things.
>>>>
>>>>
>>>>
>>>> *1.* I am trying to run one batch job with an adaptive batch
>>>> scheduler.  I referred to the elastic scaling document [1] and based on
>>>> the document I have added the below config to my FlinkDeployment. But the
>>>> job is failing with JobInitializationException: Could not start the
>>>> JobMaster. I have attached the job manager log below. *Can someone
>>>> explain why the job is failing despite using the recommended configuration?*
>>>>
>>>> jobmanager.scheduler: AdaptiveBatch
>>>> parallelism.default: "-1"
>>>> taskmanager.network.memory.buffers-per-channel: "0"
>>>> execution.batch-shuffle-mode: "ALL_EXCHANGES_BLOCKING"
>>>>
>>>>
>>>>
>>>> *2. *The document has listed some limitations of elastic scaling. One
>>>> of them is about the input file format. *My code is written in Apache
>>>> Beam, how to check, how the input is being read in Flink?*
>>>>
>>>>
>>>>    - FileInputFormat sources are not supported: FileInputFormat
>>>>    sources are not supported, including
>>>>    StreamExecutionEnvironment#readFile(...)
>>>>    StreamExecutionEnvironment#readTextFile(...) and StreamExecutionEnvironment#createInput(FileInputFormat,
>>>>    ...). Users should use the new sources(FileSystem DataStream
>>>>    Connector
>>>>    <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/filesystem/>
>>>>     or FileSystem SQL Connector
>>>>    <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/>)
>>>>    to read files when using the Adaptive Batch Scheduler.
>>>>
>>>>
>>>> Let me know if additional information is needed to debug the issue.
>>>>
>>>> [1]
>>>> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/#adaptive-batch-scheduler
>>>>
>>>> Regards
>>>> Dipak
>>>>
>>>>

Re: Batch Job with Adaptive Batch Scheduler failing with JobInitializationException: Could not start the JobMaster

Posted by Junrui Lee <jr...@gmail.com>.
Hi, Dipak

To set the execution mode to BATCH_FORCED in Flink, it's not configurable
via config option. Instead, use the Flink API with:
environment.getExecutionConfig().setExecutionMode(ExecutionMode.BATCH_FORCED);
Moreover, considering that you mentioned using Apache Beam, the
executionModeForBatch option might help you, more details can be found
here: https://beam.apache.org/documentation/runners/flink/.

Best,

Junrui

Dipak Tandel <di...@gmail.com> 于2024年3月29日周五 19:21写道:

> Hi Junrui,
>
> Thank you for bringing this to my attention. I appreciate your help in
> fixing the issue. I have gone through the latest Flink version 1.19
> documentation
> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/elastic_scaling/#limitations-2 and
> I am trying to set execution mode to BATCH_FORCED but could not figure out
> how to do it.
>
> I could not find any option to override the execution mode in the
> configuration document
> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/,
> so I added execution.mode: BATCH_FORCED config to my FlinkDeployment but
> it's not working. it is still using the default value PIPELINED, I
> validated it using the WebUI. Can you please help me figure out how to
> override the default value? What should be the value of the config key to
> override the execution mode?
>
>
>   [image: image.png]
>
> [image: image.png]
> Once again thank you for your help.
>
> Best regards,
> Dipak
>
> On Fri, Mar 29, 2024 at 2:31 PM Junrui Lee <jr...@gmail.com> wrote:
>
>> Hi Dipak,
>>
>> Regarding question 1, I noticed from the logs that the method
>> createBatchExecutionEnvironment from Beam is being used in your job. IIUC,
>> this method utilizes Flink's DataSet API. If indeed the DataSet API is
>> being used, the configuration option execution.batch-shuffle-mode will not
>> take effect, and you should set the ExecutionMode to BATCH_FORCED, in Beam
>> you can configure executionModeForBatch to BATCH_FORCED.
>> And this limitation has been documented in the latest Flink version 1.19
>> documentation:
>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/elastic_scaling/#limitations-2
>> .
>>
>> Best,
>> Junrui
>>
>> Dipak Tandel <di...@gmail.com> 于2024年3月29日周五 16:05写道:
>>
>>> Hi Everyone
>>>
>>> I am facing some issues while running the batch job on a Flink cluster
>>> using Adaptive Batch Scheduler. I have deployed a flink cluster on
>>> Kubernetes using the flink Kubernetes operator and submitted a job to the
>>> cluster using Apache beam FlinkRunner.  I am using Flink version 1.16.   I
>>> want to figure out two things.
>>>
>>>
>>>
>>> *1.* I am trying to run one batch job with an adaptive batch
>>> scheduler.  I referred to the elastic scaling document [1] and based on
>>> the document I have added the below config to my FlinkDeployment. But the
>>> job is failing with JobInitializationException: Could not start the
>>> JobMaster. I have attached the job manager log below. *Can someone
>>> explain why the job is failing despite using the recommended configuration?*
>>>
>>> jobmanager.scheduler: AdaptiveBatch
>>> parallelism.default: "-1"
>>> taskmanager.network.memory.buffers-per-channel: "0"
>>> execution.batch-shuffle-mode: "ALL_EXCHANGES_BLOCKING"
>>>
>>>
>>>
>>> *2. *The document has listed some limitations of elastic scaling. One
>>> of them is about the input file format. *My code is written in Apache
>>> Beam, how to check, how the input is being read in Flink?*
>>>
>>>
>>>    - FileInputFormat sources are not supported: FileInputFormat sources
>>>    are not supported, including StreamExecutionEnvironment#readFile(...)
>>>     StreamExecutionEnvironment#readTextFile(...) and StreamExecutionEnvironment#createInput(FileInputFormat,
>>>    ...). Users should use the new sources(FileSystem DataStream
>>>    Connector
>>>    <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/filesystem/>
>>>     or FileSystem SQL Connector
>>>    <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/>)
>>>    to read files when using the Adaptive Batch Scheduler.
>>>
>>>
>>> Let me know if additional information is needed to debug the issue.
>>>
>>> [1]
>>> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/#adaptive-batch-scheduler
>>>
>>> Regards
>>> Dipak
>>>
>>>

Re: Batch Job with Adaptive Batch Scheduler failing with JobInitializationException: Could not start the JobMaster

Posted by Dipak Tandel <di...@gmail.com>.
Hi Junrui,

Thank you for bringing this to my attention. I appreciate your help in
fixing the issue. I have gone through the latest Flink version 1.19
documentation
https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/elastic_scaling/#limitations-2
and
I am trying to set execution mode to BATCH_FORCED but could not figure out
how to do it.

I could not find any option to override the execution mode in the
configuration document
https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/,
so I added execution.mode: BATCH_FORCED config to my FlinkDeployment but
it's not working. it is still using the default value PIPELINED, I
validated it using the WebUI. Can you please help me figure out how to
override the default value? What should be the value of the config key to
override the execution mode?


  [image: image.png]

[image: image.png]
Once again thank you for your help.

Best regards,
Dipak

On Fri, Mar 29, 2024 at 2:31 PM Junrui Lee <jr...@gmail.com> wrote:

> Hi Dipak,
>
> Regarding question 1, I noticed from the logs that the method
> createBatchExecutionEnvironment from Beam is being used in your job. IIUC,
> this method utilizes Flink's DataSet API. If indeed the DataSet API is
> being used, the configuration option execution.batch-shuffle-mode will not
> take effect, and you should set the ExecutionMode to BATCH_FORCED, in Beam
> you can configure executionModeForBatch to BATCH_FORCED.
> And this limitation has been documented in the latest Flink version 1.19
> documentation:
> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/elastic_scaling/#limitations-2
> .
>
> Best,
> Junrui
>
> Dipak Tandel <di...@gmail.com> 于2024年3月29日周五 16:05写道:
>
>> Hi Everyone
>>
>> I am facing some issues while running the batch job on a Flink cluster
>> using Adaptive Batch Scheduler. I have deployed a flink cluster on
>> Kubernetes using the flink Kubernetes operator and submitted a job to the
>> cluster using Apache beam FlinkRunner.  I am using Flink version 1.16.   I
>> want to figure out two things.
>>
>>
>>
>> *1.* I am trying to run one batch job with an adaptive batch scheduler.
>> I referred to the elastic scaling document [1] and based on the document I
>> have added the below config to my FlinkDeployment. But the job is failing
>> with JobInitializationException: Could not start the JobMaster. I have
>> attached the job manager log below. *Can someone explain why the job is
>> failing despite using the recommended configuration?*
>>
>> jobmanager.scheduler: AdaptiveBatch
>> parallelism.default: "-1"
>> taskmanager.network.memory.buffers-per-channel: "0"
>> execution.batch-shuffle-mode: "ALL_EXCHANGES_BLOCKING"
>>
>>
>>
>> *2. *The document has listed some limitations of elastic scaling. One of
>> them is about the input file format. *My code is written in Apache Beam,
>> how to check, how the input is being read in Flink?*
>>
>>
>>    - FileInputFormat sources are not supported: FileInputFormat sources
>>    are not supported, including StreamExecutionEnvironment#readFile(...)
>>    StreamExecutionEnvironment#readTextFile(...) and StreamExecutionEnvironment#createInput(FileInputFormat,
>>    ...). Users should use the new sources(FileSystem DataStream Connector
>>    <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/filesystem/>
>>     or FileSystem SQL Connector
>>    <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/>)
>>    to read files when using the Adaptive Batch Scheduler.
>>
>>
>> Let me know if additional information is needed to debug the issue.
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/#adaptive-batch-scheduler
>>
>> Regards
>> Dipak
>>
>>

Re: Batch Job with Adaptive Batch Scheduler failing with JobInitializationException: Could not start the JobMaster

Posted by Junrui Lee <jr...@gmail.com>.
Hi Dipak,

Regarding question 1, I noticed from the logs that the method
createBatchExecutionEnvironment from Beam is being used in your job. IIUC,
this method utilizes Flink's DataSet API. If indeed the DataSet API is
being used, the configuration option execution.batch-shuffle-mode will not
take effect, and you should set the ExecutionMode to BATCH_FORCED, in Beam
you can configure executionModeForBatch to BATCH_FORCED.
And this limitation has been documented in the latest Flink version 1.19
documentation:
https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/elastic_scaling/#limitations-2
.

Best,
Junrui

Dipak Tandel <di...@gmail.com> 于2024年3月29日周五 16:05写道:

> Hi Everyone
>
> I am facing some issues while running the batch job on a Flink cluster
> using Adaptive Batch Scheduler. I have deployed a flink cluster on
> Kubernetes using the flink Kubernetes operator and submitted a job to the
> cluster using Apache beam FlinkRunner.  I am using Flink version 1.16.   I
> want to figure out two things.
>
>
>
> *1.* I am trying to run one batch job with an adaptive batch scheduler.
> I referred to the elastic scaling document [1] and based on the document I
> have added the below config to my FlinkDeployment. But the job is failing
> with JobInitializationException: Could not start the JobMaster. I have
> attached the job manager log below. *Can someone explain why the job is
> failing despite using the recommended configuration?*
>
> jobmanager.scheduler: AdaptiveBatch
> parallelism.default: "-1"
> taskmanager.network.memory.buffers-per-channel: "0"
> execution.batch-shuffle-mode: "ALL_EXCHANGES_BLOCKING"
>
>
>
> *2. *The document has listed some limitations of elastic scaling. One of
> them is about the input file format. *My code is written in Apache Beam,
> how to check, how the input is being read in Flink?*
>
>
>    - FileInputFormat sources are not supported: FileInputFormat sources
>    are not supported, including StreamExecutionEnvironment#readFile(...)
>    StreamExecutionEnvironment#readTextFile(...) and StreamExecutionEnvironment#createInput(FileInputFormat,
>    ...). Users should use the new sources(FileSystem DataStream Connector
>    <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/filesystem/>
>     or FileSystem SQL Connector
>    <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/>)
>    to read files when using the Adaptive Batch Scheduler.
>
>
> Let me know if additional information is needed to debug the issue.
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/#adaptive-batch-scheduler
>
> Regards
> Dipak
>
>