You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jin Yi <ji...@promoted.ai> on 2022/03/29 03:23:08 UTC
flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)
i have a flink job that uses redis as a sink. i optionally do some wiping
and metadata writing from the job submitting flink program before it
actually executes/submits the job to the job manager. when i don't do this
redis preparation, the redis sink works completely fine. that is, the
redis commands work fine from the taskmanager tasks. however, if i enable
the option redis preparation from within the flink job program, it fails to
contact redis and hits a timeout exception.
previously, we were using 1.12.3, and this behavior worked fine. is the
1.14.4 flink docker image pretty restrictive when it comes to network
access at the job submission client layer?
thanks.
Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)
Posted by 胡伟华 <hu...@gmail.com>.
Glad your issue was resolved.
> 2022年3月31日 上午12:45，Jin Yi <ji...@promoted.ai> 写道：
>
> i ended up debugging this down to a command execution timeout for the lettuce (redis client) code rather than a connection timeout. we're actually able to hit the redis server and port, but something wonky is going on w/ the redis request (command) and reply loop which is meant to be synchronous.
>
> thanks for trying to help, 胡伟华.
>
> On Tue, Mar 29, 2022 at 7:45 AM 胡伟华 <huweihua.ckl@gmail.com <ma...@gmail.com>> wrote:
> Are you referring to creating Flink cluster on Kubernetes by yaml file?
>
> How did you submit the job to Flink cluster? Not via the command line (flink run xxx)?
>
>> 2022年3月29日 下午10:38，Jin Yi <jin@promoted.ai <ma...@promoted.ai>> 写道：
>>
>> no they are not. b/c we are using k8s, we use kubectl apply commands with a yaml file to specify the startup.
>>
>> On Tue, Mar 29, 2022 at 7:37 AM 胡伟华 <huweihua.ckl@gmail.com <ma...@gmail.com>> wrote:
>> I see, can you provide the startup command for 1.12.3 and 1.14.4?
>> Are these startup commands running on the same node?
>>
>>> 2022年3月29日 下午10:32，Jin Yi <jin@promoted.ai <ma...@promoted.ai>> 写道：
>>>
>>> it's running in k8s. we're not running in app mode b/c we have many jobs running in the same flink cluster.
>>>
>>> On Tue, Mar 29, 2022 at 4:29 AM huweihua <huweihua.ckl@gmail.com <ma...@gmail.com>> wrote:
>>> Hi, Jin
>>>
>>> Can you provide more information about Flink cluster deployment modes? Is it running in Kubernetes/YARN or standalone mode?
>>> Maybe you can use application mode to keeps the environment (network accessibility) always keep same. Application mode will run the user-main method in the JobManager,
>>>
>>> [1]https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode>
>>>
>>>
>>>
>>> Jin Yi <jin@promoted.ai <ma...@promoted.ai>> 于2022年3月29日周二 11:23写道：
>>> i have a flink job that uses redis as a sink. i optionally do some wiping and metadata writing from the job submitting flink program before it actually executes/submits the job to the job manager. when i don't do this redis preparation, the redis sink works completely fine. that is, the redis commands work fine from the taskmanager tasks. however, if i enable the option redis preparation from within the flink job program, it fails to contact redis and hits a timeout exception.
>>>
>>> previously, we were using 1.12.3, and this behavior worked fine. is the 1.14.4 flink docker image pretty restrictive when it comes to network access at the job submission client layer?
>>>
>>> thanks.
>>
>
Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)
Posted by Jin Yi <ji...@promoted.ai>.
i ended up debugging this down to a command execution timeout for the
lettuce (redis client) code rather than a connection timeout. we're
actually able to hit the redis server and port, but something wonky is
going on w/ the redis request (command) and reply loop which is meant to be
synchronous.
thanks for trying to help, 胡伟华.
On Tue, Mar 29, 2022 at 7:45 AM 胡伟华 <hu...@gmail.com> wrote:
> Are you referring to creating Flink cluster on Kubernetes by yaml file?
>
> How did you submit the job to Flink cluster? Not via the command line
> (flink run xxx)?
>
> 2022年3月29日 下午10:38，Jin Yi <ji...@promoted.ai> 写道：
>
> no they are not. b/c we are using k8s, we use kubectl apply commands with
> a yaml file to specify the startup.
>
> On Tue, Mar 29, 2022 at 7:37 AM 胡伟华 <hu...@gmail.com> wrote:
>
>> I see, can you provide the startup command for 1.12.3 and 1.14.4?
>> Are these startup commands running on the same node?
>>
>> 2022年3月29日 下午10:32，Jin Yi <ji...@promoted.ai> 写道：
>>
>> it's running in k8s. we're not running in app mode b/c we have many jobs
>> running in the same flink cluster.
>>
>> On Tue, Mar 29, 2022 at 4:29 AM huweihua <hu...@gmail.com> wrote:
>>
>>> Hi, Jin
>>>
>>> Can you provide more information about Flink cluster deployment modes?
>>> Is it running in Kubernetes/YARN or standalone mode?
>>> Maybe you can use application mode to keeps the environment (network
>>> accessibility) always keep same. Application mode will run the user-main
>>> method in the JobManager,
>>>
>>> [1]
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode
>>>
>>>
>>>
>>> Jin Yi <ji...@promoted.ai> 于2022年3月29日周二 11:23写道：
>>>
>>>> i have a flink job that uses redis as a sink. i optionally do some
>>>> wiping and metadata writing from the job submitting flink program before it
>>>> actually executes/submits the job to the job manager. when i don't do this
>>>> redis preparation, the redis sink works completely fine. that is, the
>>>> redis commands work fine from the taskmanager tasks. however, if i enable
>>>> the option redis preparation from within the flink job program, it fails to
>>>> contact redis and hits a timeout exception.
>>>>
>>>> previously, we were using 1.12.3, and this behavior worked fine. is
>>>> the 1.14.4 flink docker image pretty restrictive when it comes to network
>>>> access at the job submission client layer?
>>>>
>>>> thanks.
>>>>
>>>
>>
>
Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)
Posted by 胡伟华 <hu...@gmail.com>.
Are you referring to creating Flink cluster on Kubernetes by yaml file?
How did you submit the job to Flink cluster? Not via the command line (flink run xxx)?
> 2022年3月29日 下午10:38，Jin Yi <ji...@promoted.ai> 写道：
>
> no they are not. b/c we are using k8s, we use kubectl apply commands with a yaml file to specify the startup.
>
> On Tue, Mar 29, 2022 at 7:37 AM 胡伟华 <huweihua.ckl@gmail.com <ma...@gmail.com>> wrote:
> I see, can you provide the startup command for 1.12.3 and 1.14.4?
> Are these startup commands running on the same node?
>
>> 2022年3月29日 下午10:32，Jin Yi <jin@promoted.ai <ma...@promoted.ai>> 写道：
>>
>> it's running in k8s. we're not running in app mode b/c we have many jobs running in the same flink cluster.
>>
>> On Tue, Mar 29, 2022 at 4:29 AM huweihua <huweihua.ckl@gmail.com <ma...@gmail.com>> wrote:
>> Hi, Jin
>>
>> Can you provide more information about Flink cluster deployment modes? Is it running in Kubernetes/YARN or standalone mode?
>> Maybe you can use application mode to keeps the environment (network accessibility) always keep same. Application mode will run the user-main method in the JobManager,
>>
>> [1]https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode>
>>
>>
>>
>> Jin Yi <jin@promoted.ai <ma...@promoted.ai>> 于2022年3月29日周二 11:23写道：
>> i have a flink job that uses redis as a sink. i optionally do some wiping and metadata writing from the job submitting flink program before it actually executes/submits the job to the job manager. when i don't do this redis preparation, the redis sink works completely fine. that is, the redis commands work fine from the taskmanager tasks. however, if i enable the option redis preparation from within the flink job program, it fails to contact redis and hits a timeout exception.
>>
>> previously, we were using 1.12.3, and this behavior worked fine. is the 1.14.4 flink docker image pretty restrictive when it comes to network access at the job submission client layer?
>>
>> thanks.
>
Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)
Posted by Jin Yi <ji...@promoted.ai>.
no they are not. b/c we are using k8s, we use kubectl apply commands with
a yaml file to specify the startup.
On Tue, Mar 29, 2022 at 7:37 AM 胡伟华 <hu...@gmail.com> wrote:
> I see, can you provide the startup command for 1.12.3 and 1.14.4?
> Are these startup commands running on the same node?
>
> 2022年3月29日 下午10:32，Jin Yi <ji...@promoted.ai> 写道：
>
> it's running in k8s. we're not running in app mode b/c we have many jobs
> running in the same flink cluster.
>
> On Tue, Mar 29, 2022 at 4:29 AM huweihua <hu...@gmail.com> wrote:
>
>> Hi, Jin
>>
>> Can you provide more information about Flink cluster deployment modes? Is
>> it running in Kubernetes/YARN or standalone mode?
>> Maybe you can use application mode to keeps the environment (network
>> accessibility) always keep same. Application mode will run the user-main
>> method in the JobManager,
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode
>>
>>
>>
>> Jin Yi <ji...@promoted.ai> 于2022年3月29日周二 11:23写道：
>>
>>> i have a flink job that uses redis as a sink. i optionally do some
>>> wiping and metadata writing from the job submitting flink program before it
>>> actually executes/submits the job to the job manager. when i don't do this
>>> redis preparation, the redis sink works completely fine. that is, the
>>> redis commands work fine from the taskmanager tasks. however, if i enable
>>> the option redis preparation from within the flink job program, it fails to
>>> contact redis and hits a timeout exception.
>>>
>>> previously, we were using 1.12.3, and this behavior worked fine. is the
>>> 1.14.4 flink docker image pretty restrictive when it comes to network
>>> access at the job submission client layer?
>>>
>>> thanks.
>>>
>>
>
Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)
Posted by 胡伟华 <hu...@gmail.com>.
I see, can you provide the startup command for 1.12.3 and 1.14.4?
Are these startup commands running on the same node?
> 2022年3月29日 下午10:32，Jin Yi <ji...@promoted.ai> 写道：
>
> it's running in k8s. we're not running in app mode b/c we have many jobs running in the same flink cluster.
>
> On Tue, Mar 29, 2022 at 4:29 AM huweihua <huweihua.ckl@gmail.com <ma...@gmail.com>> wrote:
> Hi, Jin
>
> Can you provide more information about Flink cluster deployment modes? Is it running in Kubernetes/YARN or standalone mode?
> Maybe you can use application mode to keeps the environment (network accessibility) always keep same. Application mode will run the user-main method in the JobManager,
>
> [1]https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode>
>
>
>
> Jin Yi <jin@promoted.ai <ma...@promoted.ai>> 于2022年3月29日周二 11:23写道：
> i have a flink job that uses redis as a sink. i optionally do some wiping and metadata writing from the job submitting flink program before it actually executes/submits the job to the job manager. when i don't do this redis preparation, the redis sink works completely fine. that is, the redis commands work fine from the taskmanager tasks. however, if i enable the option redis preparation from within the flink job program, it fails to contact redis and hits a timeout exception.
>
> previously, we were using 1.12.3, and this behavior worked fine. is the 1.14.4 flink docker image pretty restrictive when it comes to network access at the job submission client layer?
>
> thanks.
Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)
Posted by Jin Yi <ji...@promoted.ai>.
it's running in k8s. we're not running in app mode b/c we have many jobs
running in the same flink cluster.
On Tue, Mar 29, 2022 at 4:29 AM huweihua <hu...@gmail.com> wrote:
> Hi, Jin
>
> Can you provide more information about Flink cluster deployment modes? Is
> it running in Kubernetes/YARN or standalone mode?
> Maybe you can use application mode to keeps the environment (network
> accessibility) always keep same. Application mode will run the user-main
> method in the JobManager,
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode
>
>
>
> Jin Yi <ji...@promoted.ai> 于2022年3月29日周二 11:23写道：
>
>> i have a flink job that uses redis as a sink. i optionally do some
>> wiping and metadata writing from the job submitting flink program before it
>> actually executes/submits the job to the job manager. when i don't do this
>> redis preparation, the redis sink works completely fine. that is, the
>> redis commands work fine from the taskmanager tasks. however, if i enable
>> the option redis preparation from within the flink job program, it fails to
>> contact redis and hits a timeout exception.
>>
>> previously, we were using 1.12.3, and this behavior worked fine. is the
>> 1.14.4 flink docker image pretty restrictive when it comes to network
>> access at the job submission client layer?
>>
>> thanks.
>>
>
Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)
Posted by huweihua <hu...@gmail.com>.
Hi, Jin
Can you provide more information about Flink cluster deployment modes? Is
it running in Kubernetes/YARN or standalone mode?
Maybe you can use application mode to keeps the environment (network
accessibility) always keep same. Application mode will run the user-main
method in the JobManager,
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode
Jin Yi <ji...@promoted.ai> 于2022年3月29日周二 11:23写道：
> i have a flink job that uses redis as a sink. i optionally do some wiping
> and metadata writing from the job submitting flink program before it
> actually executes/submits the job to the job manager. when i don't do this
> redis preparation, the redis sink works completely fine. that is, the
> redis commands work fine from the taskmanager tasks. however, if i enable
> the option redis preparation from within the flink job program, it fails to
> contact redis and hits a timeout exception.
>
> previously, we were using 1.12.3, and this behavior worked fine. is the
> 1.14.4 flink docker image pretty restrictive when it comes to network
> access at the job submission client layer?
>
> thanks.
>