You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Alex Drobinsky <al...@gmail.com> on 2021/05/12 12:06:56 UTC
After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots
set to 1.
Dear flink community,
First I need provide some minimum information about my deployment scenario:
I'm running application inside of Flink docker, below original Dockerfile:
-----------------------------------------------------------------------------------------------------------
FROM flink:1.13.0-scala_2.11-java11
# Copy log and monitoring related JARs to flink lib dir
COPY kafka-clients-2.4.1.jar /opt/flink/lib/
RUN chmod 777 /tmp
RUN apt-get update && apt-get install -y htop
# configuration files
COPY Log4cpp.properties /opt/flink/
COPY Log4j.properties /opt/flink/conf/log4j.properties
COPY SessionOrganizer.json /opt/flink/
*COPY flink-conf.yaml /opt/flink/conf/COPY slaves /opt/flink/conf/*
# job file
COPY KafkaToSessions-shade.jar /opt/flink/lib/
# libraries
ADD libs /usr/local/lib/
# Add /usr/local/lib to ldconfig
RUN echo "/usr/local/lib/" > /etc/ld.so.conf.d/ips.conf && \
ldconfig && \
ulimit -c 0
RUN mkdir /opt/flink/ip-collection/ && \
mkdir /opt/flink/checkpoints/ && \
mkdir /opt/flink/ip-collection/incorrectIcs && \
mkdir /opt/flink/ip-collection/storage && \
mkdir /opt/flink/ip-collection/logs
CMD /opt/flink/bin/start-cluster.sh && /opt/flink/bin/flink run
/opt/flink/lib/KafkaToSessions-shade.jar
-------------------------------------------------------------------------------------------------------------------------------
If we will ignore irrelevant parts of Dockerfile, the only 2 things remains
( beside FROM statement)
1. Overwritten flink-conf.yml + slaves
2. CMD which executes start-cluster and job.
My flink-conf.yml:
---------------------------------------------------------------------------------------------------------
rest.address: localhost
rest.port: 8088
state.backend: filesystem
state.checkpoints.dir: file:///opt/flink/checkpoints
jobmanager.memory.process.size: 2224m
jobmanager.rpc.port: 6123
jobmanager.rpc.address: localhost
taskmanager.memory.flink.size: 2224m
taskmanager.memory.task.heap.size: 1000m
taskmanager.numberOfTaskSlots: 12
taskmanager.rpc.port: 50100
taskmanager.data.port: 50000
parallelism.default: 6
heartbeat.timeout: 120000
heartbeat.interval: 20000
env.java.opts: "-XX:+UseG1GC -XX:MaxGCPauseMillis=300"
---------------------------------------------------------------------------------------------------------
Slaves file contain single line with localhost.
After start of docker, I noticed that application doesn't work due lack of
slots. When I checked flink-conf.yml I noticed that
taskmanager.numberOfTaskSlots is set to 1.
P.S. during first time, daemon.sh complained that it doesn't have write
permissions to change flink-conf.yml, when I added chown flink.flink
/opt/flink/flink-conf.yml -
it stopped to complain & taskmanager.numberOfTaskSlots change occured.
Any suggestions ?
Best regards,
Alexander
Re: After upgrade from 1.11.2 to 1.13.0 parameter
taskmanager.numberOfTaskSlots set to 1.
Posted by Alexey Trenikhun <ye...@msn.com>.
If flink-conf.yaml is readonly, flink will complain but work fine?
________________________________
From: Chesnay Schepler <ch...@apache.org>
Sent: Wednesday, May 12, 2021 5:38 AM
To: Alex Drobinsky <al...@gmail.com>
Cc: user@flink.apache.org <us...@flink.apache.org>
Subject: Re: After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots set to 1.
The contents of FLINK_PROPERTIES are piped as-is into the flink configuration, and thus require the same format as the configuration.
On 5/12/2021 2:36 PM, Alex Drobinsky wrote:
Thanks a lot !
I used TASK_MANAGER_NUMBER_OF_TASK_SLOTS in my docker-compose.yml, it works perfectly :)
In which format I could provide parameters via FLINK_PROPERTIES ? I'm thinking of abandoning the idea to copy flink-conf in Dockerfile.
Is it limited to a specific set of parameters or generic ?
ср, 12 мая 2021 г. в 15:20, Chesnay Schepler <ch...@apache.org>>:
You could also configure the number of slots via the TASK_MANAGER_NUMBER_OF_TASK_SLOTS environment variable.
On 5/12/2021 2:19 PM, Chesnay Schepler wrote:
I believe this is due to FLINK-21037; we did not consider the possibility of users mounting the configuration directly, and instead assumed that modifications to the config always go through the FLINK_PROPERTIES environment variable.
That would also be the workaround for your issue.
On 5/12/2021 2:06 PM, Alex Drobinsky wrote:
Dear flink community,
First I need provide some minimum information about my deployment scenario:
I'm running application inside of Flink docker, below original Dockerfile:
-----------------------------------------------------------------------------------------------------------
FROM flink:1.13.0-scala_2.11-java11
# Copy log and monitoring related JARs to flink lib dir
COPY kafka-clients-2.4.1.jar /opt/flink/lib/
RUN chmod 777 /tmp
RUN apt-get update && apt-get install -y htop
# configuration files
COPY Log4cpp.properties /opt/flink/
COPY Log4j.properties /opt/flink/conf/log4j.properties
COPY SessionOrganizer.json /opt/flink/
COPY flink-conf.yaml /opt/flink/conf/
COPY slaves /opt/flink/conf/
# job file
COPY KafkaToSessions-shade.jar /opt/flink/lib/
# libraries
ADD libs /usr/local/lib/
# Add /usr/local/lib to ldconfig
RUN echo "/usr/local/lib/" > /etc/ld.so.conf.d/ips.conf && \
ldconfig && \
ulimit -c 0
RUN mkdir /opt/flink/ip-collection/ && \
mkdir /opt/flink/checkpoints/ && \
mkdir /opt/flink/ip-collection/incorrectIcs && \
mkdir /opt/flink/ip-collection/storage && \
mkdir /opt/flink/ip-collection/logs
CMD /opt/flink/bin/start-cluster.sh && /opt/flink/bin/flink run /opt/flink/lib/KafkaToSessions-shade.jar
-------------------------------------------------------------------------------------------------------------------------------
If we will ignore irrelevant parts of Dockerfile, the only 2 things remains ( beside FROM statement)
1. Overwritten flink-conf.yml + slaves
2. CMD which executes start-cluster and job.
My flink-conf.yml:
---------------------------------------------------------------------------------------------------------
rest.address: localhost
rest.port: 8088
state.backend: filesystem
state.checkpoints.dir: file:///opt/flink/checkpoints
jobmanager.memory.process.size: 2224m
jobmanager.rpc.port: 6123
jobmanager.rpc.address: localhost
taskmanager.memory.flink.size: 2224m
taskmanager.memory.task.heap.size: 1000m
taskmanager.numberOfTaskSlots: 12
taskmanager.rpc.port: 50100
taskmanager.data.port: 50000
parallelism.default: 6
heartbeat.timeout: 120000
heartbeat.interval: 20000
env.java.opts: "-XX:+UseG1GC -XX:MaxGCPauseMillis=300"
---------------------------------------------------------------------------------------------------------
Slaves file contain single line with localhost.
After start of docker, I noticed that application doesn't work due lack of slots. When I checked flink-conf.yml I noticed that taskmanager.numberOfTaskSlots is set to 1.
P.S. during first time, daemon.sh complained that it doesn't have write permissions to change flink-conf.yml, when I added chown flink.flink /opt/flink/flink-conf.yml -
it stopped to complain & taskmanager.numberOfTaskSlots change occured.
Any suggestions ?
Best regards,
Alexander
Re: After upgrade from 1.11.2 to 1.13.0 parameter
taskmanager.numberOfTaskSlots set to 1.
Posted by Chesnay Schepler <ch...@apache.org>.
The contents of FLINK_PROPERTIES are piped as-is into the flink
configuration, and thus require the same format as the configuration.
On 5/12/2021 2:36 PM, Alex Drobinsky wrote:
> Thanks a lot !
> I used TASK_MANAGER_NUMBER_OF_TASK_SLOTS in my docker-compose.yml, it
> works perfectly :)
> In which format I could provide parameters via FLINK_PROPERTIES ? I'm
> thinking of abandoning the idea to copy flink-conf in Dockerfile.
> Is it limited to a specific set of parameters or generic ?
>
> ср, 12 мая 2021 г. в 15:20, Chesnay Schepler <chesnay@apache.org
> <ma...@apache.org>>:
>
> You could also configure the number of slots via the
> TASK_MANAGER_NUMBER_OF_TASK_SLOTS environment variable.
>
> On 5/12/2021 2:19 PM, Chesnay Schepler wrote:
>> I believe this is due to FLINK-21037; we did not consider the
>> possibility of users mounting the configuration directly, and
>> instead assumed that modifications to the config always go
>> through the FLINK_PROPERTIES environment variable.
>>
>> That would also be the workaround for your issue.
>>
>> On 5/12/2021 2:06 PM, Alex Drobinsky wrote:
>>> Dear flink community,
>>>
>>> First I need provide some minimum information about my
>>> deployment scenario:
>>> I'm running application inside of Flink docker, below original
>>> Dockerfile:
>>> -----------------------------------------------------------------------------------------------------------
>>> FROM flink:1.13.0-scala_2.11-java11
>>>
>>> # Copy log and monitoring related JARs to flink lib dir
>>> COPY kafka-clients-2.4.1.jar /opt/flink/lib/
>>>
>>> RUN chmod 777 /tmp
>>> RUN apt-get update && apt-get install -y htop
>>>
>>> # configuration files
>>> COPY Log4cpp.properties /opt/flink/
>>> COPY Log4j.properties /opt/flink/conf/log4j.properties
>>> COPY SessionOrganizer.json /opt/flink/
>>> *COPY flink-conf.yaml /opt/flink/conf/ COPY slaves /opt/flink/conf/ *
>>> # job file
>>> COPY KafkaToSessions-shade.jar /opt/flink/lib/
>>>
>>> # libraries
>>> ADD libs /usr/local/lib/
>>>
>>> # Add /usr/local/lib to ldconfig
>>> RUN echo "/usr/local/lib/" > /etc/ld.so.conf.d/ips.conf && \
>>> ldconfig && \
>>> ulimit -c 0
>>>
>>> RUN mkdir /opt/flink/ip-collection/ && \
>>> mkdir /opt/flink/checkpoints/ && \
>>> mkdir /opt/flink/ip-collection/incorrectIcs && \
>>> mkdir /opt/flink/ip-collection/storage && \
>>> mkdir /opt/flink/ip-collection/logs
>>>
>>> CMD /opt/flink/bin/start-cluster.sh && /opt/flink/bin/flink run /opt/flink/lib/KafkaToSessions-shade.jar
>>> -------------------------------------------------------------------------------------------------------------------------------
>>> If we will ignore irrelevant parts of Dockerfile, the only 2
>>> things remains ( beside FROM statement)
>>> 1. Overwritten flink-conf.yml + slaves
>>> 2. CMD which executes start-cluster and job.
>>>
>>> My flink-conf.yml:
>>> ---------------------------------------------------------------------------------------------------------
>>> rest.address: localhost
>>> rest.port: 8088
>>> state.backend: filesystem
>>> state.checkpoints.dir:file:///opt/flink/checkpoints
>>> jobmanager.memory.process.size: 2224m
>>> jobmanager.rpc.port: 6123
>>> jobmanager.rpc.address: localhost
>>> taskmanager.memory.flink.size: 2224m
>>> taskmanager.memory.task.heap.size: 1000m
>>> taskmanager.numberOfTaskSlots: 12
>>> taskmanager.rpc.port: 50100
>>> taskmanager.data.port: 50000
>>> parallelism.default: 6
>>> heartbeat.timeout: 120000
>>> heartbeat.interval: 20000
>>> env.java.opts:"-XX:+UseG1GC -XX:MaxGCPauseMillis=300"
>>> ---------------------------------------------------------------------------------------------------------
>>> Slaves file contain single line with localhost.
>>> After start of docker, I noticed that application doesn't work
>>> due lack of slots. When I checked flink-conf.yml I noticed that
>>> taskmanager.numberOfTaskSlots is set to 1.
>>> P.S. during first time, daemon.sh complained that it doesn't
>>> have write permissions to change flink-conf.yml, when I added
>>> chown flink.flink /opt/flink/flink-conf.yml -
>>> it stopped to complain & taskmanager.numberOfTaskSlots change
>>> occured.
>>>
>>> Any suggestions ?
>>>
>>> Best regards,
>>> Alexander
>>>
>>>
>>
>
Re: After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots
set to 1.
Posted by Alex Drobinsky <al...@gmail.com>.
Thanks a lot !
I used TASK_MANAGER_NUMBER_OF_TASK_SLOTS in my docker-compose.yml, it works
perfectly :)
In which format I could provide parameters via FLINK_PROPERTIES ? I'm
thinking of abandoning the idea to copy flink-conf in Dockerfile.
Is it limited to a specific set of parameters or generic ?
ср, 12 мая 2021 г. в 15:20, Chesnay Schepler <ch...@apache.org>:
> You could also configure the number of slots via the TASK_MANAGER_NUMBER_OF_TASK_SLOTS
> environment variable.
>
> On 5/12/2021 2:19 PM, Chesnay Schepler wrote:
>
> I believe this is due to FLINK-21037; we did not consider the possibility
> of users mounting the configuration directly, and instead assumed that
> modifications to the config always go through the FLINK_PROPERTIES
> environment variable.
>
> That would also be the workaround for your issue.
>
> On 5/12/2021 2:06 PM, Alex Drobinsky wrote:
>
> Dear flink community,
>
> First I need provide some minimum information about my deployment scenario:
> I'm running application inside of Flink docker, below original Dockerfile:
>
> -----------------------------------------------------------------------------------------------------------
>
> FROM flink:1.13.0-scala_2.11-java11
>
> # Copy log and monitoring related JARs to flink lib dir
> COPY kafka-clients-2.4.1.jar /opt/flink/lib/
>
> RUN chmod 777 /tmp
> RUN apt-get update && apt-get install -y htop
>
> # configuration files
> COPY Log4cpp.properties /opt/flink/
> COPY Log4j.properties /opt/flink/conf/log4j.properties
> COPY SessionOrganizer.json /opt/flink/*COPY flink-conf.yaml /opt/flink/conf/
> COPY slaves /opt/flink/conf/
> *
> # job file
> COPY KafkaToSessions-shade.jar /opt/flink/lib/
>
> # libraries
> ADD libs /usr/local/lib/
>
> # Add /usr/local/lib to ldconfig
> RUN echo "/usr/local/lib/" > /etc/ld.so.conf.d/ips.conf && \
> ldconfig && \
> ulimit -c 0
>
> RUN mkdir /opt/flink/ip-collection/ && \
> mkdir /opt/flink/checkpoints/ && \
> mkdir /opt/flink/ip-collection/incorrectIcs && \
> mkdir /opt/flink/ip-collection/storage && \
> mkdir /opt/flink/ip-collection/logs
>
> CMD /opt/flink/bin/start-cluster.sh && /opt/flink/bin/flink run /opt/flink/lib/KafkaToSessions-shade.jar
>
>
> -------------------------------------------------------------------------------------------------------------------------------
> If we will ignore irrelevant parts of Dockerfile, the only 2 things
> remains ( beside FROM statement)
> 1. Overwritten flink-conf.yml + slaves
> 2. CMD which executes start-cluster and job.
>
> My flink-conf.yml:
>
> ---------------------------------------------------------------------------------------------------------
>
> rest.address: localhostrest.port: 8088state.backend: filesystemstate.checkpoints.dir: file:///opt/flink/checkpointsjobmanager.memory.process.size: 2224mjobmanager.rpc.port: 6123jobmanager.rpc.address: localhosttaskmanager.memory.flink.size: 2224mtaskmanager.memory.task.heap.size: 1000mtaskmanager.numberOfTaskSlots: 12taskmanager.rpc.port: 50100taskmanager.data.port: 50000parallelism.default: 6heartbeat.timeout: 120000heartbeat.interval: 20000env.java.opts: "-XX:+UseG1GC -XX:MaxGCPauseMillis=300"
>
> ---------------------------------------------------------------------------------------------------------
>
> Slaves file contain single line with localhost.
> After start of docker, I noticed that application doesn't work due lack of
> slots. When I checked flink-conf.yml I noticed that
> taskmanager.numberOfTaskSlots is set to 1.
> P.S. during first time, daemon.sh complained that it doesn't have write
> permissions to change flink-conf.yml, when I added chown flink.flink
> /opt/flink/flink-conf.yml -
> it stopped to complain & taskmanager.numberOfTaskSlots change occured.
>
> Any suggestions ?
>
> Best regards,
> Alexander
>
>
>
>
>
Re: After upgrade from 1.11.2 to 1.13.0 parameter
taskmanager.numberOfTaskSlots set to 1.
Posted by Chesnay Schepler <ch...@apache.org>.
You could also configure the number of slots via the
TASK_MANAGER_NUMBER_OF_TASK_SLOTS environment variable.
On 5/12/2021 2:19 PM, Chesnay Schepler wrote:
> I believe this is due to FLINK-21037; we did not consider the
> possibility of users mounting the configuration directly, and instead
> assumed that modifications to the config always go through the
> FLINK_PROPERTIES environment variable.
>
> That would also be the workaround for your issue.
>
> On 5/12/2021 2:06 PM, Alex Drobinsky wrote:
>> Dear flink community,
>>
>> First I need provide some minimum information about my deployment
>> scenario:
>> I'm running application inside of Flink docker, below original
>> Dockerfile:
>> -----------------------------------------------------------------------------------------------------------
>> FROM flink:1.13.0-scala_2.11-java11
>>
>> # Copy log and monitoring related JARs to flink lib dir
>> COPY kafka-clients-2.4.1.jar /opt/flink/lib/
>>
>> RUN chmod 777 /tmp
>> RUN apt-get update && apt-get install -y htop
>>
>> # configuration files
>> COPY Log4cpp.properties /opt/flink/
>> COPY Log4j.properties /opt/flink/conf/log4j.properties
>> COPY SessionOrganizer.json /opt/flink/
>> *COPY flink-conf.yaml /opt/flink/conf/ COPY slaves /opt/flink/conf/ *
>> # job file
>> COPY KafkaToSessions-shade.jar /opt/flink/lib/
>>
>> # libraries
>> ADD libs /usr/local/lib/
>>
>> # Add /usr/local/lib to ldconfig
>> RUN echo "/usr/local/lib/" > /etc/ld.so.conf.d/ips.conf && \
>> ldconfig && \
>> ulimit -c 0
>>
>> RUN mkdir /opt/flink/ip-collection/ && \
>> mkdir /opt/flink/checkpoints/ && \
>> mkdir /opt/flink/ip-collection/incorrectIcs && \
>> mkdir /opt/flink/ip-collection/storage && \
>> mkdir /opt/flink/ip-collection/logs
>>
>> CMD /opt/flink/bin/start-cluster.sh && /opt/flink/bin/flink run /opt/flink/lib/KafkaToSessions-shade.jar
>> -------------------------------------------------------------------------------------------------------------------------------
>> If we will ignore irrelevant parts of Dockerfile, the only 2 things
>> remains ( beside FROM statement)
>> 1. Overwritten flink-conf.yml + slaves
>> 2. CMD which executes start-cluster and job.
>>
>> My flink-conf.yml:
>> ---------------------------------------------------------------------------------------------------------
>> rest.address: localhost
>> rest.port: 8088
>> state.backend: filesystem
>> state.checkpoints.dir:file:///opt/flink/checkpoints
>> jobmanager.memory.process.size: 2224m
>> jobmanager.rpc.port: 6123
>> jobmanager.rpc.address: localhost
>> taskmanager.memory.flink.size: 2224m
>> taskmanager.memory.task.heap.size: 1000m
>> taskmanager.numberOfTaskSlots: 12
>> taskmanager.rpc.port: 50100
>> taskmanager.data.port: 50000
>> parallelism.default: 6
>> heartbeat.timeout: 120000
>> heartbeat.interval: 20000
>> env.java.opts:"-XX:+UseG1GC -XX:MaxGCPauseMillis=300"
>> ---------------------------------------------------------------------------------------------------------
>> Slaves file contain single line with localhost.
>> After start of docker, I noticed that application doesn't work due
>> lack of slots. When I checked flink-conf.yml I noticed that
>> taskmanager.numberOfTaskSlots is set to 1.
>> P.S. during first time, daemon.sh complained that it doesn't have
>> write permissions to change flink-conf.yml, when I added chown
>> flink.flink /opt/flink/flink-conf.yml -
>> it stopped to complain & taskmanager.numberOfTaskSlots change occured.
>>
>> Any suggestions ?
>>
>> Best regards,
>> Alexander
>>
>>
>
Re: After upgrade from 1.11.2 to 1.13.0 parameter
taskmanager.numberOfTaskSlots set to 1.
Posted by Chesnay Schepler <ch...@apache.org>.
I believe this is due to FLINK-21037; we did not consider the
possibility of users mounting the configuration directly, and instead
assumed that modifications to the config always go through the
FLINK_PROPERTIES environment variable.
That would also be the workaround for your issue.
On 5/12/2021 2:06 PM, Alex Drobinsky wrote:
> Dear flink community,
>
> First I need provide some minimum information about my deployment
> scenario:
> I'm running application inside of Flink docker, below original Dockerfile:
> -----------------------------------------------------------------------------------------------------------
> FROM flink:1.13.0-scala_2.11-java11
>
> # Copy log and monitoring related JARs to flink lib dir
> COPY kafka-clients-2.4.1.jar /opt/flink/lib/
>
> RUN chmod 777 /tmp
> RUN apt-get update && apt-get install -y htop
>
> # configuration files
> COPY Log4cpp.properties /opt/flink/
> COPY Log4j.properties /opt/flink/conf/log4j.properties
> COPY SessionOrganizer.json /opt/flink/
> *COPY flink-conf.yaml /opt/flink/conf/ COPY slaves /opt/flink/conf/ *
> # job file
> COPY KafkaToSessions-shade.jar /opt/flink/lib/
>
> # libraries
> ADD libs /usr/local/lib/
>
> # Add /usr/local/lib to ldconfig
> RUN echo "/usr/local/lib/" > /etc/ld.so.conf.d/ips.conf && \
> ldconfig && \
> ulimit -c 0
>
> RUN mkdir /opt/flink/ip-collection/ && \
> mkdir /opt/flink/checkpoints/ && \
> mkdir /opt/flink/ip-collection/incorrectIcs && \
> mkdir /opt/flink/ip-collection/storage && \
> mkdir /opt/flink/ip-collection/logs
>
> CMD /opt/flink/bin/start-cluster.sh && /opt/flink/bin/flink run /opt/flink/lib/KafkaToSessions-shade.jar
> -------------------------------------------------------------------------------------------------------------------------------
> If we will ignore irrelevant parts of Dockerfile, the only 2 things
> remains ( beside FROM statement)
> 1. Overwritten flink-conf.yml + slaves
> 2. CMD which executes start-cluster and job.
>
> My flink-conf.yml:
> ---------------------------------------------------------------------------------------------------------
> rest.address: localhost
> rest.port: 8088
> state.backend: filesystem
> state.checkpoints.dir: file:///opt/flink/checkpoints
> jobmanager.memory.process.size: 2224m
> jobmanager.rpc.port: 6123
> jobmanager.rpc.address: localhost
> taskmanager.memory.flink.size: 2224m
> taskmanager.memory.task.heap.size: 1000m
> taskmanager.numberOfTaskSlots: 12
> taskmanager.rpc.port: 50100
> taskmanager.data.port: 50000
> parallelism.default: 6
> heartbeat.timeout: 120000
> heartbeat.interval: 20000
> env.java.opts:"-XX:+UseG1GC -XX:MaxGCPauseMillis=300"
> ---------------------------------------------------------------------------------------------------------
> Slaves file contain single line with localhost.
> After start of docker, I noticed that application doesn't work due
> lack of slots. When I checked flink-conf.yml I noticed that
> taskmanager.numberOfTaskSlots is set to 1.
> P.S. during first time, daemon.sh complained that it doesn't have
> write permissions to change flink-conf.yml, when I added chown
> flink.flink /opt/flink/flink-conf.yml -
> it stopped to complain & taskmanager.numberOfTaskSlots change occured.
>
> Any suggestions ?
>
> Best regards,
> Alexander
>
>