You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Jinzhong Li <li...@gmail.com> on 2024/03/07 08:52:47 UTC
[DISCUSS] FLIP-426: Grouping Remote State Access
Hi devs,
I'd like to start a discussion on a sub-FLIP of FLIP-423: Disaggregated
State Storage and Management[1], which is a joint work of Yuan Mei, Zakelly
Lan, Jinzhong Li, Hangxiang Yu, Yanfei Lei and Feng Wang:
- FLIP-426: Grouping Remote State Access
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-426%3A+Grouping+Remote+State+Access>
[2]
This FLIP enables retrieval of remote state data in batches to avoid
unnecessary round-trip costs for remote access.
Please make sure you have read the FLIP-423[1] to know the whole story, and
we'll discuss the details of FLIP-424[2] under this mail. For the
discussion of overall architecture or topics related with multiple
sub-FLIPs, please post in the previous mail[3].
Looking forward to hearing from you!
[1] https://cwiki.apache.org/confluence/x/R4p3EQ
[2] https://cwiki.apache.org/confluence/x/TYp3EQ
[3] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
Best,
Jinzhong Li
Re: [DISCUSS] FLIP-426: Grouping Remote State Access
Posted by Jinzhong Li <li...@gmail.com>.
Hi jeyhun,
Thanks for your feedback!
>> Do we have a fallback mechanism for filesystems that do not support
multiget?
Currently, the rocksdb multiGet is available only for PosixFileSystem. And
the multiGet requires the filesystem to support async read interfaces (e.g.
MultiRead, ReadAsync, poll, abort).
We plan to support multiGet on FlinkFileSystem in ForSt by proxying all
async interfaces to corresponding methods of ForStFlieSystem (introduced in
FLIP-427[1]).
And I think we could introduce fallback mechanism for the ForStFlieSystem
that don't support these async interface. (e.g. using synchronous read
instead of asynchronous read.)
>> Also, in the case of multiget what is the granularity of error handling
or retry semantics (e.g., one subrequest fails in multiget). Do we fully
rely on RocksDB's semantics in this case?
For exceptions caused by remote file system access, I think we could
introduce a unified error handling or retry mechanism in ForStFlieSystem[1]
interfaces which is also useful for ForSt's other put and get interface.
For other exceptions, we can rely on ForSt's mulitget semantics.
>> How do we plan to separate/allocate the resource utilization of this
component? Because AEC receives (potentially many) requests from
operators, it can easily be a bottleneck to the whole pipeline.
The AEC shares both CPU and IO resources with other components within the
TM. To be honest, in scenarios of disaggregated state storage, it is the
remote IO access that tends to be the bottleneck, not the CPU operations of
the AEC. A significant portion of the TM's network IO resources will be
consumed by remote state access, and the throughput of these remote IO
access has a direct impact on the performance of the whole pipeline.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-427%3A+ForSt+-+Disaggregated+state+Store
Best,
Jinzhong
On Sat, Mar 9, 2024 at 11:56 PM Jeyhun Karimov <je...@gmail.com> wrote:
> Hi Jinzhong,
>
> +1 for the FLIP. I have the following comments:
>
> - Do we have a fallback mechanism for filesystems that do not support
> multiget?
>
> - Also, in the case of multiget what is the granularity of error handling
> or retry semantics (e.g., one subrequest fails in multiget). Do we fully
> rely on RocksDB's semantics in this case?
>
> - From my understanding AEC and especially its grouping component performs
> computationally expensive operations.
> Basically, operators offload async requests to AEC, and it performs
> serialization/ordering/optimizations on these requests before submitting
> them.
> How do we plan to separate/allocate the resource utilization of this
> component? Because AEC receives (potentially many) requests from operators,
> it can easily be a bottleneck to the whole pipeline.
>
> Regards,
> Jeyhun
>
> On Thu, Mar 7, 2024 at 9:53 AM Jinzhong Li <li...@gmail.com>
> wrote:
>
> > Hi devs,
> >
> >
> > I'd like to start a discussion on a sub-FLIP of FLIP-423: Disaggregated
> > State Storage and Management[1], which is a joint work of Yuan Mei,
> Zakelly
> > Lan, Jinzhong Li, Hangxiang Yu, Yanfei Lei and Feng Wang:
> >
> > - FLIP-426: Grouping Remote State Access
> > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-426%3A+Grouping+Remote+State+Access
> > >
> > [2]
> >
> > This FLIP enables retrieval of remote state data in batches to avoid
> > unnecessary round-trip costs for remote access.
> >
> > Please make sure you have read the FLIP-423[1] to know the whole story,
> and
> > we'll discuss the details of FLIP-424[2] under this mail. For the
> > discussion of overall architecture or topics related with multiple
> > sub-FLIPs, please post in the previous mail[3].
> >
> > Looking forward to hearing from you!
> >
> > [1] https://cwiki.apache.org/confluence/x/R4p3EQ
> >
> > [2] https://cwiki.apache.org/confluence/x/TYp3EQ
> >
> > [3] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
> >
> > Best,
> >
> > Jinzhong Li
> >
>
Re: [DISCUSS] FLIP-426: Grouping Remote State Access
Posted by Jeyhun Karimov <je...@gmail.com>.
Hi Jinzhong,
+1 for the FLIP. I have the following comments:
- Do we have a fallback mechanism for filesystems that do not support
multiget?
- Also, in the case of multiget what is the granularity of error handling
or retry semantics (e.g., one subrequest fails in multiget). Do we fully
rely on RocksDB's semantics in this case?
- From my understanding AEC and especially its grouping component performs
computationally expensive operations.
Basically, operators offload async requests to AEC, and it performs
serialization/ordering/optimizations on these requests before submitting
them.
How do we plan to separate/allocate the resource utilization of this
component? Because AEC receives (potentially many) requests from operators,
it can easily be a bottleneck to the whole pipeline.
Regards,
Jeyhun
On Thu, Mar 7, 2024 at 9:53 AM Jinzhong Li <li...@gmail.com> wrote:
> Hi devs,
>
>
> I'd like to start a discussion on a sub-FLIP of FLIP-423: Disaggregated
> State Storage and Management[1], which is a joint work of Yuan Mei, Zakelly
> Lan, Jinzhong Li, Hangxiang Yu, Yanfei Lei and Feng Wang:
>
> - FLIP-426: Grouping Remote State Access
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-426%3A+Grouping+Remote+State+Access
> >
> [2]
>
> This FLIP enables retrieval of remote state data in batches to avoid
> unnecessary round-trip costs for remote access.
>
> Please make sure you have read the FLIP-423[1] to know the whole story, and
> we'll discuss the details of FLIP-424[2] under this mail. For the
> discussion of overall architecture or topics related with multiple
> sub-FLIPs, please post in the previous mail[3].
>
> Looking forward to hearing from you!
>
> [1] https://cwiki.apache.org/confluence/x/R4p3EQ
>
> [2] https://cwiki.apache.org/confluence/x/TYp3EQ
>
> [3] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
>
> Best,
>
> Jinzhong Li
>
Re: [DISCUSS] FLIP-426: Grouping Remote State Access
Posted by Jinzhong Li <li...@gmail.com>.
Hi Yue,
Thanks for your comments.
I get your point.I think there are two possible ways we can support
multiGet in synchronization models in future:
1) Implement the multiGet interface in the existing Rocksdb State
interface, then integrate this interface to the synchronous Operators layer.
2) Integrate the ForStStateBackend interface to synchronous model/operators.
No matter which approach we will choose in the future, I don't think this
flip's proposal will be a block on extending the use of multiGet on
synchronous models. And although ForStStateBackend is first used for
asynchronous models, it will preserve the extensibility used by the
synchronization model. We don't need to worry too much about this :-)
Best,
Jinzhong
On Wed, Mar 20, 2024 at 4:37 PM yue ma <ma...@gmail.com> wrote:
> hi Jinzhong
> Thanks for your reply
> The reason why I mentioned this point is because according to the official
> Rocksdb documentation
> https://rocksdb.org/blog/2022/10/07/asynchronous-io-in-rocksdb.html. if we
> turn on async_io and use multiGet, it can improve the performance of point
> look upc by 100%. Moreover, especially in Flink SQL tasks, there are many
> ways to access state through mini batch, so I believe this feature also
> greatly optimizes the synchronous access method and is worth doing.
> If we first support batch access for asynchronous models, I think it would
> be okay. My point is, should we consider whether it can be easily extended
> if we support synchronous models in the future
>
> Jinzhong Li <li...@gmail.com> 于2024年3月19日周二 20:59写道：
>
> > Hi Yue,
> >
> > Thanks for your feedback!
> >
> > > 1. Does Grouping Remote State Access only support asynchronous
> > interfaces?
> > > --If it is: IIUC, MultiGet can also greatly improve performance for
> > > synchronous access modes. Do we need to support it ？
> >
> > Yes. If we want to support MultiGet on existing synchronous access mode,
> we
> > have to introduce a grouping component akin to the AEC described in
> > FLIP-425[1].
> > I think such a change would introduce additional complexity to the
> current
> > synchronous model, and the extent of performance gains remains uncertain.
> > Therefore, I recommend only asynchronous interfaces support "Grouping
> > Remote State Access", which is designed to efficiently minimize latency
> in
> > accessing remote state storage.
> >
> > > 2. Can a simple example be added to FLip on how to use Batch to access
> > > states and obtain the results of states on the API?
> >
> > Sure. I have added a code example in the Flip[2]. Note that the multiget
> in
> > this Flip is an internal interface, not a user-facing interface.
> >
> > > 3. I also agree with XiaoRui's viewpoint. Is there a corresponding
> Config
> > > to control the state access batch strategy?
> >
> > Yes, we would offer some configurable options that allow users to adjust
> > the behavior of batching and grouping state access (eg. batching size,
> > etc.).
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-425%3A+Asynchronous+Execution+Model
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-426%3A+Grouping+Remote+State+Access#FLIP426:GroupingRemoteStateAccess-CodeExampleonHowtoAccessStateUsingBatch
> >
> > Best,
> > Jinzhong Li
> >
> >
> > On Tue, Mar 19, 2024 at 5:52 PM yue ma <ma...@gmail.com> wrote:
> >
> > > Hi Jinzhong,
> > >
> > > Thanks for the FLIP. I have the following questions:
> > >
> > > 1. Does Grouping Remote State Access only support asynchronous
> > interfaces?
> > > --If it is: IIUC, MultiGet can also greatly improve performance for
> > > synchronous access modes. Do we need to support it ？
> > > --If not, how can we distinguish between using Grouping State
> Access
> > in
> > > asynchronous and synchronous modes?
> > > 2. Can a simple example be added to FLip on how to use Batch to access
> > > states and obtain the results of states on the API?
> > > 3. I also agree with XiaoRui's viewpoint. Is there a corresponding
> Config
> > > to control the state access batch strategy?
> > >
> > > --
> > > Best,
> > > Yue
> > >
> >
>
>
> --
> Best,
> Yue
>
Re: [DISCUSS] FLIP-426: Grouping Remote State Access
Posted by yue ma <ma...@gmail.com>.
hi Jinzhong
Thanks for your reply
The reason why I mentioned this point is because according to the official
Rocksdb documentation
https://rocksdb.org/blog/2022/10/07/asynchronous-io-in-rocksdb.html. if we
turn on async_io and use multiGet, it can improve the performance of point
look upc by 100%. Moreover, especially in Flink SQL tasks, there are many
ways to access state through mini batch, so I believe this feature also
greatly optimizes the synchronous access method and is worth doing.
If we first support batch access for asynchronous models, I think it would
be okay. My point is, should we consider whether it can be easily extended
if we support synchronous models in the future
Jinzhong Li <li...@gmail.com> 于2024年3月19日周二 20:59写道：
> Hi Yue,
>
> Thanks for your feedback!
>
> > 1. Does Grouping Remote State Access only support asynchronous
> interfaces?
> > --If it is: IIUC, MultiGet can also greatly improve performance for
> > synchronous access modes. Do we need to support it ？
>
> Yes. If we want to support MultiGet on existing synchronous access mode, we
> have to introduce a grouping component akin to the AEC described in
> FLIP-425[1].
> I think such a change would introduce additional complexity to the current
> synchronous model, and the extent of performance gains remains uncertain.
> Therefore, I recommend only asynchronous interfaces support "Grouping
> Remote State Access", which is designed to efficiently minimize latency in
> accessing remote state storage.
>
> > 2. Can a simple example be added to FLip on how to use Batch to access
> > states and obtain the results of states on the API?
>
> Sure. I have added a code example in the Flip[2]. Note that the multiget in
> this Flip is an internal interface, not a user-facing interface.
>
> > 3. I also agree with XiaoRui's viewpoint. Is there a corresponding Config
> > to control the state access batch strategy?
>
> Yes, we would offer some configurable options that allow users to adjust
> the behavior of batching and grouping state access (eg. batching size,
> etc.).
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-425%3A+Asynchronous+Execution+Model
> [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-426%3A+Grouping+Remote+State+Access#FLIP426:GroupingRemoteStateAccess-CodeExampleonHowtoAccessStateUsingBatch
>
> Best,
> Jinzhong Li
>
>
> On Tue, Mar 19, 2024 at 5:52 PM yue ma <ma...@gmail.com> wrote:
>
> > Hi Jinzhong,
> >
> > Thanks for the FLIP. I have the following questions:
> >
> > 1. Does Grouping Remote State Access only support asynchronous
> interfaces?
> > --If it is: IIUC, MultiGet can also greatly improve performance for
> > synchronous access modes. Do we need to support it ？
> > --If not, how can we distinguish between using Grouping State Access
> in
> > asynchronous and synchronous modes?
> > 2. Can a simple example be added to FLip on how to use Batch to access
> > states and obtain the results of states on the API?
> > 3. I also agree with XiaoRui's viewpoint. Is there a corresponding Config
> > to control the state access batch strategy?
> >
> > --
> > Best,
> > Yue
> >
>
--
Best,
Yue
Re: [DISCUSS] FLIP-426: Grouping Remote State Access
Posted by Jinzhong Li <li...@gmail.com>.
Hi everyone,
Thanks for your valuable discussion and feedback!
Our discussions have been going on for a while. If there are no more
concerns, I would like to start the vote thread after 72 hours,thanks again!
Please let me know if you have any concerns, thanks!
Best,
Jinzhong
On Tue, Mar 19, 2024 at 8:56 PM Jinzhong Li <li...@gmail.com>
wrote:
> Hi Yue,
>
> Thanks for your feedback!
>
> > 1. Does Grouping Remote State Access only support asynchronous
> interfaces?
> > --If it is: IIUC, MultiGet can also greatly improve performance for
> > synchronous access modes. Do we need to support it ？
>
> Yes. If we want to support MultiGet on existing synchronous access mode,
> we have to introduce a grouping component akin to the AEC described in
> FLIP-425[1].
> I think such a change would introduce additional complexity to the current
> synchronous model, and the extent of performance gains remains uncertain.
> Therefore, I recommend only asynchronous interfaces support "Grouping
> Remote State Access", which is designed to efficiently minimize latency in
> accessing remote state storage.
>
> > 2. Can a simple example be added to FLip on how to use Batch to access
> > states and obtain the results of states on the API?
>
> Sure. I have added a code example in the Flip[2]. Note that the multiget
> in this Flip is an internal interface, not a user-facing interface.
>
> > 3. I also agree with XiaoRui's viewpoint. Is there a corresponding Config
> > to control the state access batch strategy?
>
> Yes, we would offer some configurable options that allow users to adjust
> the behavior of batching and grouping state access (eg. batching size,
> etc.).
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-425%3A+Asynchronous+Execution+Model
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-426%3A+Grouping+Remote+State+Access#FLIP426:GroupingRemoteStateAccess-CodeExampleonHowtoAccessStateUsingBatch
>
> Best,
> Jinzhong Li
>
>
> On Tue, Mar 19, 2024 at 5:52 PM yue ma <ma...@gmail.com> wrote:
>
>> Hi Jinzhong,
>>
>> Thanks for the FLIP. I have the following questions:
>>
>> 1. Does Grouping Remote State Access only support asynchronous interfaces?
>> --If it is: IIUC, MultiGet can also greatly improve performance for
>> synchronous access modes. Do we need to support it ？
>> --If not, how can we distinguish between using Grouping State Access
>> in
>> asynchronous and synchronous modes?
>> 2. Can a simple example be added to FLip on how to use Batch to access
>> states and obtain the results of states on the API?
>> 3. I also agree with XiaoRui's viewpoint. Is there a corresponding Config
>> to control the state access batch strategy?
>>
>> --
>> Best,
>> Yue
>>
>
Re: [DISCUSS] FLIP-426: Grouping Remote State Access
Posted by Jinzhong Li <li...@gmail.com>.
Hi Yue,
Thanks for your feedback!
> 1. Does Grouping Remote State Access only support asynchronous interfaces?
> --If it is: IIUC, MultiGet can also greatly improve performance for
> synchronous access modes. Do we need to support it ？
Yes. If we want to support MultiGet on existing synchronous access mode, we
have to introduce a grouping component akin to the AEC described in
FLIP-425[1].
I think such a change would introduce additional complexity to the current
synchronous model, and the extent of performance gains remains uncertain.
Therefore, I recommend only asynchronous interfaces support "Grouping
Remote State Access", which is designed to efficiently minimize latency in
accessing remote state storage.
> 2. Can a simple example be added to FLip on how to use Batch to access
> states and obtain the results of states on the API?
Sure. I have added a code example in the Flip[2]. Note that the multiget in
this Flip is an internal interface, not a user-facing interface.
> 3. I also agree with XiaoRui's viewpoint. Is there a corresponding Config
> to control the state access batch strategy?
Yes, we would offer some configurable options that allow users to adjust
the behavior of batching and grouping state access (eg. batching size,
etc.).
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-425%3A+Asynchronous+Execution+Model
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-426%3A+Grouping+Remote+State+Access#FLIP426:GroupingRemoteStateAccess-CodeExampleonHowtoAccessStateUsingBatch
Best,
Jinzhong Li
On Tue, Mar 19, 2024 at 5:52 PM yue ma <ma...@gmail.com> wrote:
> Hi Jinzhong,
>
> Thanks for the FLIP. I have the following questions:
>
> 1. Does Grouping Remote State Access only support asynchronous interfaces?
> --If it is: IIUC, MultiGet can also greatly improve performance for
> synchronous access modes. Do we need to support it ？
> --If not, how can we distinguish between using Grouping State Access in
> asynchronous and synchronous modes?
> 2. Can a simple example be added to FLip on how to use Batch to access
> states and obtain the results of states on the API?
> 3. I also agree with XiaoRui's viewpoint. Is there a corresponding Config
> to control the state access batch strategy?
>
> --
> Best,
> Yue
>
Re: [DISCUSS] FLIP-426: Grouping Remote State Access
Posted by yue ma <ma...@gmail.com>.
Hi Jinzhong,
Thanks for the FLIP. I have the following questions:
1. Does Grouping Remote State Access only support asynchronous interfaces?
--If it is: IIUC, MultiGet can also greatly improve performance for
synchronous access modes. Do we need to support it ？
--If not, how can we distinguish between using Grouping State Access in
asynchronous and synchronous modes?
2. Can a simple example be added to FLip on how to use Batch to access
states and obtain the results of states on the API?
3. I also agree with XiaoRui's viewpoint. Is there a corresponding Config
to control the state access batch strategy?
--
Best,
Yue
Re: [DISCUSS] FLIP-426: Grouping Remote State Access
Posted by Jinzhong Li <li...@gmail.com>.
Hi Rui,
Thanks for your comments.
> In my opinion, we need to reduce the times of fetching RocksDB
> SST from remote to local. The FLIP seems to batch the RocksDB
> put/get requests. I am not sure this will reduce the SST fetching times.
For multiple Get(read) requests, we will convert them into one MultiGet
request. The MultiGet could merge block read IO which belong to multiple
Get requests, thereby reducing the times of fetching RocksDB Block/SST from
remote filesystem.
> How to monitor the I/O used by state disaggregation?
On the one hand, the remote storage(S3/HDFS/OSS) generally provides
network/IO throughput monitoring. On the other hand, on the Flink side we
can also provide some metric about accessing remote storage, eg. the ratio
of remote storage access to local disk cache access, etc.
> How about customizing batching strategy? I
I don't recommend permitting users to customize their batching strategy, as
very few users have the capability to customize this strategy effectively.
However, we could offer some configurable options that allow users to
adjust the behavior of batching, such as the batching size and so on.
Best,
Jinzhong
On Fri, Mar 15, 2024 at 12:08 PM 夏 瑞 <xi...@hotmail.com> wrote:
> Hi Jinzhong,
>
> Batching state access is a reasonable way to reduce the amount of I/O
> compared to per-record state access. But I have some questions:
>
> - In my opinion, we need to reduce the times of fetching RocksDB SST from
> remote to local. The FLIP seems to batch the RocksDB put/get requests. I am
> not sure this will reduce the SST fetching times.
>
> - How to monitor the I/O used by state disaggregation? The latency/amount
> of I/O on DFS is important to the performance diagnosis. Moreover, the
> amount of I/O also influences the stability of the DFS. For example, the
> throughput of HDFS NameNode is hard to scale and suffers from I/O flood.
>
> - How about customizing batching strategy? Intuitively, an extremely large
> batch may need lots of memory to hold the returned results, and causes OOM.
> On the other size, if we randomly batch keys, state storage may navigate
> all SSTs to find the results.
>
> Best wishes,
> Rui Xia.
>
> ________________________________
> 发件人: Hangxiang Yu <ma...@gmail.com>
> 发送时间: 2024年3月15日 4:04
> 收件人: xiarui0428@hotmail.com <xi...@hotmail.com>
> 主题: Fwd: [DISCUSS] FLIP-426: Grouping Remote State Access
>
>
>
> ---------- Forwarded message ---------
> From: Jinzhong Li <lijinzhong2051@gmail.com<mailto:
> lijinzhong2051@gmail.com>>
> Date: Thu, Mar 7, 2024 at 4:52 PM
> Subject: [DISCUSS] FLIP-426: Grouping Remote State Access
> To: <de...@flink.apache.org>>
> Cc: <yu...@gmail.com>>, <
> zakelly.lan@gmail.com<ma...@gmail.com>>, <masteryhx@gmail.com
> <ma...@gmail.com>>, <fredialei@gmail.com<mailto:
> fredialei@gmail.com>>, <fe...@apache.org>>
>
>
>
> Hi devs,
>
>
> I'd like to start a discussion on a sub-FLIP of FLIP-423: Disaggregated
> State Storage and Management[1], which is a joint work of Yuan Mei, Zakelly
> Lan, Jinzhong Li, Hangxiang Yu, Yanfei Lei and Feng Wang:
>
> - FLIP-426: Grouping Remote State Access<
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-426%3A+Grouping+Remote+State+Access>
> [2]
>
> This FLIP enables retrieval of remote state data in batches to avoid
> unnecessary round-trip costs for remote access.
>
> Please make sure you have read the FLIP-423[1] to know the whole story,
> and we'll discuss the details of FLIP-424[2] under this mail. For the
> discussion of overall architecture or topics related with multiple
> sub-FLIPs, please post in the previous mail[3].
>
> Looking forward to hearing from you!
>
> [1] https://cwiki.apache.org/confluence/x/R4p3EQ
>
> [2] https://cwiki.apache.org/confluence/x/TYp3EQ
>
> [3] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
>
> Best,
>
> Jinzhong Li
>
>
>
>
> --
> Best,
> Hangxiang.
>
回复: [DISCUSS] FLIP-426: Grouping Remote State Access
Posted by 夏 瑞 <xi...@hotmail.com>.
Hi Jinzhong,
Batching state access is a reasonable way to reduce the amount of I/O compared to per-record state access. But I have some questions:
- In my opinion, we need to reduce the times of fetching RocksDB SST from remote to local. The FLIP seems to batch the RocksDB put/get requests. I am not sure this will reduce the SST fetching times.
- How to monitor the I/O used by state disaggregation? The latency/amount of I/O on DFS is important to the performance diagnosis. Moreover, the amount of I/O also influences the stability of the DFS. For example, the throughput of HDFS NameNode is hard to scale and suffers from I/O flood.
- How about customizing batching strategy? Intuitively, an extremely large batch may need lots of memory to hold the returned results, and causes OOM. On the other size, if we randomly batch keys, state storage may navigate all SSTs to find the results.
Best wishes,
Rui Xia.
________________________________
发件人: Hangxiang Yu <ma...@gmail.com>
发送时间: 2024年3月15日 4:04
收件人: xiarui0428@hotmail.com <xi...@hotmail.com>
主题: Fwd: [DISCUSS] FLIP-426: Grouping Remote State Access
---------- Forwarded message ---------
From: Jinzhong Li <li...@gmail.com>>
Date: Thu, Mar 7, 2024 at 4:52 PM
Subject: [DISCUSS] FLIP-426: Grouping Remote State Access
To: <de...@flink.apache.org>>
Cc: <yu...@gmail.com>>, <za...@gmail.com>>, <ma...@gmail.com>>, <fr...@gmail.com>>, <fe...@apache.org>>
Hi devs,
I'd like to start a discussion on a sub-FLIP of FLIP-423: Disaggregated State Storage and Management[1], which is a joint work of Yuan Mei, Zakelly Lan, Jinzhong Li, Hangxiang Yu, Yanfei Lei and Feng Wang:
- FLIP-426: Grouping Remote State Access<https://cwiki.apache.org/confluence/display/FLINK/FLIP-426%3A+Grouping+Remote+State+Access> [2]
This FLIP enables retrieval of remote state data in batches to avoid unnecessary round-trip costs for remote access.
Please make sure you have read the FLIP-423[1] to know the whole story, and we'll discuss the details of FLIP-424[2] under this mail. For the discussion of overall architecture or topics related with multiple sub-FLIPs, please post in the previous mail[3].
Looking forward to hearing from you!
[1] https://cwiki.apache.org/confluence/x/R4p3EQ
[2] https://cwiki.apache.org/confluence/x/TYp3EQ
[3] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
Best,
Jinzhong Li
--
Best,
Hangxiang.