You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "tsreaper (via GitHub)" <gi...@apache.org> on 2023/05/31 06:34:49 UTC
[GitHub] [incubator-paimon] tsreaper commented on a diff in pull request #1266: [Feature] Add docs about managed memory in flink.md
tsreaper commented on code in PR #1266:
URL: https://github.com/apache/incubator-paimon/pull/1266#discussion_r1211149100
##########
docs/content/engines/flink.md:
##########
@@ -210,3 +210,19 @@ All Flink data types are supported, except that
* `MULTISET` is not supported.
* `MAP` is not supported as primary keys.
+
+## Use Flink Managed Memory
+In order to improve the buffer pool for writers in Paimon tasks.Paimon tasks can create memory pools based on Executor Memory which will be managed by Executor, such as Managed Memory in Flink TaskManager. It will improve the stability and performance of sinks by managing writer buffers for multiple tasks through Executor.
+
+The following properties can be set if using the flink managed memory:
Review Comment:
```suggestion
The following properties can be set if using Flink managed memory:
```
##########
docs/content/engines/flink.md:
##########
@@ -210,3 +210,19 @@ All Flink data types are supported, except that
* `MULTISET` is not supported.
* `MAP` is not supported as primary keys.
+
+## Use Flink Managed Memory
+In order to improve the buffer pool for writers in Paimon tasks.Paimon tasks can create memory pools based on Executor Memory which will be managed by Executor, such as Managed Memory in Flink TaskManager. It will improve the stability and performance of sinks by managing writer buffers for multiple tasks through Executor.
+
+The following properties can be set if using the flink managed memory:
+
+| Option | Default | Description |
+|------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| sink.use-managed-memory-allocator | false | If true, flink sink will use managed memory for merge tree; otherwise, it will create an independent memory allocator. |
Review Comment:
```suggestion
| sink.use-managed-memory-allocator | false | If true, Flink sink will use managed memory for merge tree; otherwise, it will create an independent memory allocator. |
```
Also, what does "independent memory allocator" mean? An independent memory allocator for each parallelism? You might also want to stress that this allocator will use heap memory, not Flink managed memory.
##########
docs/content/engines/flink.md:
##########
@@ -210,3 +210,19 @@ All Flink data types are supported, except that
* `MULTISET` is not supported.
* `MAP` is not supported as primary keys.
+
+## Use Flink Managed Memory
+In order to improve the buffer pool for writers in Paimon tasks.Paimon tasks can create memory pools based on Executor Memory which will be managed by Executor, such as Managed Memory in Flink TaskManager. It will improve the stability and performance of sinks by managing writer buffers for multiple tasks through Executor.
Review Comment:
```suggestion
Paimon tasks can create memory pools based on executor memory which will be managed by Flink executor, such as managed memory in Flink task manager. It will improve the stability and performance of sinks by managing writer buffers for multiple tasks through executor.
```
##########
docs/content/engines/flink.md:
##########
@@ -210,3 +210,19 @@ All Flink data types are supported, except that
* `MULTISET` is not supported.
* `MAP` is not supported as primary keys.
+
+## Use Flink Managed Memory
+In order to improve the buffer pool for writers in Paimon tasks.Paimon tasks can create memory pools based on Executor Memory which will be managed by Executor, such as Managed Memory in Flink TaskManager. It will improve the stability and performance of sinks by managing writer buffers for multiple tasks through Executor.
+
+The following properties can be set if using the flink managed memory:
+
+| Option | Default | Description |
+|------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| sink.use-managed-memory-allocator | false | If true, flink sink will use managed memory for merge tree; otherwise, it will create an independent memory allocator. |
+| sink.managed.writer-buffer-memory | 256M | Weight of writer buffer in managed memory, Flink will compute the memory size, for writer according to the weight, the actual memory used depends on the running environment. |
Review Comment:
"Weight of writer buffer"? This is not the real memory size? Could you provide a link to Flink's document about how managed memory are allocated to different operators?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org