You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Xiaolong Wang <xi...@smartnews.com> on 2021/09/10 06:29:13 UTC
How to mount PVC volumes using Flink Native Kubernetes ?
Hi,
I'm facing a tough question. I want to start a Flink Native Kubernetes
job with each of the task manager pod mounted with an aws-ebs PVC.
The first thought is to use the pod-template file to do this, but it soon
went to a dead end. Since the pod-template on each of the task manager pod
is the same, how can I mount different PVCs ?
This issue is quite puzzling, will you please help me ?
Thanks in advance !
Re: How to mount PVC volumes using Flink Native Kubernetes ?
Posted by Yang Wang <da...@gmail.com>.
Thanks Tao for providing your internal use case.
I have create a ticket for this feature[1].
[1]. https://issues.apache.org/jira/browse/FLINK-24332
Best,
Yang
tao xiao <xi...@gmail.com> 于2021年9月11日周六 上午10:18写道：
> Thanks David for the tips. We have been running Flink with no performance
> degradation observed in EMR (which is EBS attached) for more than 1 year
> therefore we believe the same performance can be applied in Kubernetes.
>
> On Sat, Sep 11, 2021 at 3:13 AM David Morávek <dm...@apache.org> wrote:
>
>> OT: Beware that even if you manage to solve this, EBS is replicated
>> network storage, therefore rocksdb performance will be affected
>> significantly.
>>
>> Best,
>> D.
>>
>> On Fri 10. 9. 2021 at 16:19, tao xiao <xi...@gmail.com> wrote:
>>
>>> The use case we have is to store the RocksDB sst files in EBS. The EC2
>>> instance type (m5) we use doesn't provide local disk storage therefore EBS
>>> is the only option to store the local sst file.
>>>
>>> On Fri, Sep 10, 2021 at 7:10 PM Yang Wang <da...@gmail.com> wrote:
>>>
>>>> I am afraid Flink could not support creating dedicated PVC for each
>>>> TaskManager pod now.
>>>> But I think it might be a reasonable requirement.
>>>>
>>>> Could you please share why you need to mount a persistent volume claim
>>>> per TaskManager?
>>>> AFAIK, the TaskManager will be deleted once it fails. You expect the
>>>> PVC to also be deleted. Right?
>>>>
>>>>
>>>> Best,
>>>> Yang
>>>>
>>>> Xiaolong Wang <xi...@smartnews.com> 于2021年9月10日周五 下午2:37写道：
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm facing a tough question. I want to start a Flink Native
>>>>> Kubernetes job with each of the task manager pod mounted with an aws-ebs
>>>>> PVC.
>>>>>
>>>>> The first thought is to use the pod-template file to do this, but it
>>>>> soon went to a dead end. Since the pod-template on each of the task manager
>>>>> pod is the same, how can I mount different PVCs ?
>>>>>
>>>>> This issue is quite puzzling, will you please help me ?
>>>>>
>>>>> Thanks in advance !
>>>>>
>>>>
>>>
>>> --
>>> Regards,
>>> Tao
>>>
>>
>
> --
> Regards,
> Tao
>
Re: How to mount PVC volumes using Flink Native Kubernetes ?
Posted by tao xiao <xi...@gmail.com>.
Thanks David for the tips. We have been running Flink with no performance
degradation observed in EMR (which is EBS attached) for more than 1 year
therefore we believe the same performance can be applied in Kubernetes.
On Sat, Sep 11, 2021 at 3:13 AM David Morávek <dm...@apache.org> wrote:
> OT: Beware that even if you manage to solve this, EBS is replicated
> network storage, therefore rocksdb performance will be affected
> significantly.
>
> Best,
> D.
>
> On Fri 10. 9. 2021 at 16:19, tao xiao <xi...@gmail.com> wrote:
>
>> The use case we have is to store the RocksDB sst files in EBS. The EC2
>> instance type (m5) we use doesn't provide local disk storage therefore EBS
>> is the only option to store the local sst file.
>>
>> On Fri, Sep 10, 2021 at 7:10 PM Yang Wang <da...@gmail.com> wrote:
>>
>>> I am afraid Flink could not support creating dedicated PVC for each
>>> TaskManager pod now.
>>> But I think it might be a reasonable requirement.
>>>
>>> Could you please share why you need to mount a persistent volume claim
>>> per TaskManager?
>>> AFAIK, the TaskManager will be deleted once it fails. You expect the PVC
>>> to also be deleted. Right?
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> Xiaolong Wang <xi...@smartnews.com> 于2021年9月10日周五 下午2:37写道：
>>>
>>>> Hi,
>>>>
>>>> I'm facing a tough question. I want to start a Flink Native
>>>> Kubernetes job with each of the task manager pod mounted with an aws-ebs
>>>> PVC.
>>>>
>>>> The first thought is to use the pod-template file to do this, but it
>>>> soon went to a dead end. Since the pod-template on each of the task manager
>>>> pod is the same, how can I mount different PVCs ?
>>>>
>>>> This issue is quite puzzling, will you please help me ?
>>>>
>>>> Thanks in advance !
>>>>
>>>
>>
>> --
>> Regards,
>> Tao
>>
>
--
Regards,
Tao
Re: How to mount PVC volumes using Flink Native Kubernetes ?
Posted by David Morávek <dm...@apache.org>.
OT: Beware that even if you manage to solve this, EBS is replicated network
storage, therefore rocksdb performance will be affected significantly.
Best,
D.
On Fri 10. 9. 2021 at 16:19, tao xiao <xi...@gmail.com> wrote:
> The use case we have is to store the RocksDB sst files in EBS. The EC2
> instance type (m5) we use doesn't provide local disk storage therefore EBS
> is the only option to store the local sst file.
>
> On Fri, Sep 10, 2021 at 7:10 PM Yang Wang <da...@gmail.com> wrote:
>
>> I am afraid Flink could not support creating dedicated PVC for each
>> TaskManager pod now.
>> But I think it might be a reasonable requirement.
>>
>> Could you please share why you need to mount a persistent volume claim
>> per TaskManager?
>> AFAIK, the TaskManager will be deleted once it fails. You expect the PVC
>> to also be deleted. Right?
>>
>>
>> Best,
>> Yang
>>
>> Xiaolong Wang <xi...@smartnews.com> 于2021年9月10日周五 下午2:37写道：
>>
>>> Hi,
>>>
>>> I'm facing a tough question. I want to start a Flink Native
>>> Kubernetes job with each of the task manager pod mounted with an aws-ebs
>>> PVC.
>>>
>>> The first thought is to use the pod-template file to do this, but it
>>> soon went to a dead end. Since the pod-template on each of the task manager
>>> pod is the same, how can I mount different PVCs ?
>>>
>>> This issue is quite puzzling, will you please help me ?
>>>
>>> Thanks in advance !
>>>
>>
>
> --
> Regards,
> Tao
>
Re: How to mount PVC volumes using Flink Native Kubernetes ?
Posted by tao xiao <xi...@gmail.com>.
The use case we have is to store the RocksDB sst files in EBS. The EC2
instance type (m5) we use doesn't provide local disk storage therefore EBS
is the only option to store the local sst file.
On Fri, Sep 10, 2021 at 7:10 PM Yang Wang <da...@gmail.com> wrote:
> I am afraid Flink could not support creating dedicated PVC for each
> TaskManager pod now.
> But I think it might be a reasonable requirement.
>
> Could you please share why you need to mount a persistent volume claim per
> TaskManager?
> AFAIK, the TaskManager will be deleted once it fails. You expect the PVC
> to also be deleted. Right?
>
>
> Best,
> Yang
>
> Xiaolong Wang <xi...@smartnews.com> 于2021年9月10日周五 下午2:37写道：
>
>> Hi,
>>
>> I'm facing a tough question. I want to start a Flink Native Kubernetes
>> job with each of the task manager pod mounted with an aws-ebs PVC.
>>
>> The first thought is to use the pod-template file to do this, but it
>> soon went to a dead end. Since the pod-template on each of the task manager
>> pod is the same, how can I mount different PVCs ?
>>
>> This issue is quite puzzling, will you please help me ?
>>
>> Thanks in advance !
>>
>
--
Regards,
Tao
Re: How to mount PVC volumes using Flink Native Kubernetes ?
Posted by Yang Wang <da...@gmail.com>.
I am afraid Flink could not support creating dedicated PVC for each
TaskManager pod now.
But I think it might be a reasonable requirement.
Could you please share why you need to mount a persistent volume claim per
TaskManager?
AFAIK, the TaskManager will be deleted once it fails. You expect the PVC to
also be deleted. Right?
Best,
Yang
Xiaolong Wang <xi...@smartnews.com> 于2021年9月10日周五 下午2:37写道：
> Hi,
>
> I'm facing a tough question. I want to start a Flink Native Kubernetes
> job with each of the task manager pod mounted with an aws-ebs PVC.
>
> The first thought is to use the pod-template file to do this, but it
> soon went to a dead end. Since the pod-template on each of the task manager
> pod is the same, how can I mount different PVCs ?
>
> This issue is quite puzzling, will you please help me ?
>
> Thanks in advance !
>