You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/01/08 08:51:25 UTC
[GitHub] [flink] JingsongLi opened a new pull request #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
JingsongLi opened a new pull request #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797
## What is the purpose of the change
Now after FLINK-14063 , operators will get all manage memory of TaskManager, The cost of over allocate memory is very high, lead to performance regression of small batch sql jobs:
Allocate memory will have the cost of memory management algorithm.
Allocate memory will have the cost of memory initialization, will set all memory to zero. And this initialization will require the operating system to actually allocate physical memory.
Over allocate memory will squash the file cache too.
We can optimize the operator algorithm, apply lazy allocation, and avoid meaningless memory allocation.
## Brief change log
- Introduce LazyMemorySegmentPool
- Improve operators to lazily allocate memory
## Verifying this change
This change is already covered by existing tests.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): (no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no
- The S3 file system connector: (no
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? JavaDocs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/143532669 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
Hash:5628de7fba5004f793d14a16fff55186ce7f3d41 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144087946 TriggerType:PUSH TriggerID:5628de7fba5004f793d14a16fff55186ce7f3d41
Hash:5628de7fba5004f793d14a16fff55186ce7f3d41 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277 TriggerType:PUSH TriggerID:5628de7fba5004f793d14a16fff55186ce7f3d41
-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187)
* 5628de7fba5004f793d14a16fff55186ce7f3d41 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144087946) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] JingsongLi closed pull request #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
JingsongLi closed pull request #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187",
"triggerID" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"triggerType" : "PUSH"
}, {
"hash" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"status" : "FAILURE",
"url" : "https://travis-ci.com/flink-ci/flink/builds/143532669",
"triggerID" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"triggerType" : "PUSH"
}, {
"hash" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/flink-ci/flink/builds/144087946",
"triggerID" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"triggerType" : "PUSH"
}, {
"hash" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277",
"triggerID" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"triggerType" : "PUSH"
}, {
"hash" : "a32248db5be831164281fff2448a346c91645817",
"status" : "PENDING",
"url" : "https://travis-ci.com/flink-ci/flink/builds/150197408",
"triggerID" : "a32248db5be831164281fff2448a346c91645817",
"triggerType" : "PUSH"
}, {
"hash" : "a32248db5be831164281fff2448a346c91645817",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5483",
"triggerID" : "a32248db5be831164281fff2448a346c91645817",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187)
* 5628de7fba5004f793d14a16fff55186ce7f3d41 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144087946) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277)
* a32248db5be831164281fff2448a346c91645817 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/150197408) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5483)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187",
"triggerID" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"triggerType" : "PUSH"
}, {
"hash" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/143532669",
"triggerID" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"triggerType" : "PUSH"
}, {
"hash" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"status" : "DELETED",
"url" : "https://travis-ci.com/flink-ci/flink/builds/144087946",
"triggerID" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"triggerType" : "PUSH"
}, {
"hash" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277",
"triggerID" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"triggerType" : "PUSH"
}, {
"hash" : "a32248db5be831164281fff2448a346c91645817",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/flink-ci/flink/builds/150197408",
"triggerID" : "a32248db5be831164281fff2448a346c91645817",
"triggerType" : "PUSH"
}, {
"hash" : "a32248db5be831164281fff2448a346c91645817",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5483",
"triggerID" : "a32248db5be831164281fff2448a346c91645817",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a32248db5be831164281fff2448a346c91645817 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/150197408) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5483)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/143532669 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TsReaper commented on a change in pull request #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
TsReaper commented on a change in pull request #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#discussion_r365085550
##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/util/LazyMemorySegmentPool.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.util;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.memory.MemoryAllocationException;
+import org.apache.flink.runtime.memory.MemoryManager;
+
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * {@link MemorySegmentPool} that lazy allocate memory pages from {@link MemoryManager}.
+ */
+public class LazyMemorySegmentPool implements MemorySegmentPool, Closeable {
+
+ private static final long PER_REQUEST_MEMORY_SIZE = 16 * 1024 * 1024;
+
+ private final Object owner;
+ private final MemoryManager memoryManager;
+ private final boolean freeMore;
+ private final ArrayList<MemorySegment> cachePages;
+ private final int maxPages;
+ private final int perRequestPages;
+
+ private int pageUsage;
+
+ public LazyMemorySegmentPool(Object owner, MemoryManager memoryManager, int maxPages) {
+ this(owner, memoryManager, maxPages, false);
+ }
+
+ public LazyMemorySegmentPool(Object owner, MemoryManager memoryManager, int maxPages, boolean freeMore) {
+ this.owner = owner;
+ this.memoryManager = memoryManager;
+ this.freeMore = freeMore;
+ this.cachePages = new ArrayList<>();
+ this.maxPages = maxPages;
+ this.pageUsage = 0;
+ this.perRequestPages = Math.max(1, (int) (PER_REQUEST_MEMORY_SIZE / memoryManager.getPageSize()));
+ }
+
+ @Override
+ public int pageSize() {
+ return this.memoryManager.getPageSize();
+ }
+
+ @Override
+ public void returnAll(List<MemorySegment> memory) {
+ this.pageUsage -= memory.size();
+ if (this.pageUsage < 0) {
+ throw new RuntimeException("Return too more memories.");
+ }
+ this.cachePages.addAll(memory);
+
+ if (this.freeMore && this.cachePages.size() >= this.perRequestPages) {
+ this.memoryManager.release(this.cachePages);
+ }
+ }
+
+ public void returnPage(MemorySegment segment) {
+ returnAll(Collections.singletonList(segment));
+ }
+
+ @Override
+ public MemorySegment nextSegment() {
+ int freePages = freePages();
+ if (freePages == 0) {
+ return null;
+ }
+
+ if (this.cachePages.isEmpty()) {
+ int numPages = Math.min(freePages, this.perRequestPages);
+ try {
+ this.memoryManager.allocatePages(owner, this.cachePages, numPages);
+ } catch (MemoryAllocationException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ this.pageUsage++;
+ return this.cachePages.remove(this.cachePages.size() - 1);
+ }
+
+ @Override
+ public int freePages() {
+ return this.maxPages - this.pageUsage;
+ }
+
+ @Override
+ public void close() {
+ if (this.pageUsage != 0) {
+ throw new RuntimeException(
+ "Should return all used memory before clean, page used: " + pageUsage);
+ }
+ cleanCache();
+ }
+
+ public void cleanCache() {
+ this.memoryManager.release(this.cachePages);
Review comment:
Shall we check if `this.cachePages.size() >= this.perRequestPages`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571951479
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit e904ab518c4d3b26bf27a89c61b9df30af54b6ed (Wed Jan 08 08:54:28 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187",
"triggerID" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"triggerType" : "PUSH"
}, {
"hash" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"status" : "FAILURE",
"url" : "https://travis-ci.com/flink-ci/flink/builds/143532669",
"triggerID" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"triggerType" : "PUSH"
}, {
"hash" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/flink-ci/flink/builds/144087946",
"triggerID" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"triggerType" : "PUSH"
}, {
"hash" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277",
"triggerID" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"triggerType" : "PUSH"
}, {
"hash" : "a32248db5be831164281fff2448a346c91645817",
"status" : "PENDING",
"url" : "https://travis-ci.com/flink-ci/flink/builds/150197408",
"triggerID" : "a32248db5be831164281fff2448a346c91645817",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187)
* 5628de7fba5004f793d14a16fff55186ce7f3d41 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144087946) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277)
* a32248db5be831164281fff2448a346c91645817 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/150197408)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187",
"triggerID" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"triggerType" : "PUSH"
}, {
"hash" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"status" : "FAILURE",
"url" : "https://travis-ci.com/flink-ci/flink/builds/143532669",
"triggerID" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"triggerType" : "PUSH"
}, {
"hash" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/flink-ci/flink/builds/144087946",
"triggerID" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"triggerType" : "PUSH"
}, {
"hash" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277",
"triggerID" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187)
* 5628de7fba5004f793d14a16fff55186ce7f3d41 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144087946) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TsReaper commented on a change in pull request #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
TsReaper commented on a change in pull request #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#discussion_r365084839
##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/sort/BinaryExternalSorter.java
##########
@@ -1054,7 +1018,7 @@ private void releaseEmptyBuffers() {
}
}
}
- releaseCoreSegments();
+ sortReadMemory.forEach(LazyMemorySegmentPool::cleanCache);
Review comment:
We don't need to manually clean cache, as memory pool will try to clean cache when a segment is returned.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187",
"triggerID" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"triggerType" : "PUSH"
}, {
"hash" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"status" : "FAILURE",
"url" : "https://travis-ci.com/flink-ci/flink/builds/143532669",
"triggerID" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"triggerType" : "PUSH"
}, {
"hash" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/flink-ci/flink/builds/144087946",
"triggerID" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"triggerType" : "PUSH"
}, {
"hash" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277",
"triggerID" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"triggerType" : "PUSH"
}, {
"hash" : "a32248db5be831164281fff2448a346c91645817",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/flink-ci/flink/builds/150197408",
"triggerID" : "a32248db5be831164281fff2448a346c91645817",
"triggerType" : "PUSH"
}, {
"hash" : "a32248db5be831164281fff2448a346c91645817",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5483",
"triggerID" : "a32248db5be831164281fff2448a346c91645817",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187)
* 5628de7fba5004f793d14a16fff55186ce7f3d41 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144087946) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277)
* a32248db5be831164281fff2448a346c91645817 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/150197408) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5483)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/143532669 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
Hash:5628de7fba5004f793d14a16fff55186ce7f3d41 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:5628de7fba5004f793d14a16fff55186ce7f3d41
-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187)
* 5628de7fba5004f793d14a16fff55186ce7f3d41 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] JingsongLi commented on a change in pull request #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on a change in pull request #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#discussion_r365086087
##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/util/LazyMemorySegmentPool.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.util;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.memory.MemoryAllocationException;
+import org.apache.flink.runtime.memory.MemoryManager;
+
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * {@link MemorySegmentPool} that lazy allocate memory pages from {@link MemoryManager}.
+ */
+public class LazyMemorySegmentPool implements MemorySegmentPool, Closeable {
+
+ private static final long PER_REQUEST_MEMORY_SIZE = 16 * 1024 * 1024;
+
+ private final Object owner;
+ private final MemoryManager memoryManager;
+ private final boolean freeMore;
+ private final ArrayList<MemorySegment> cachePages;
+ private final int maxPages;
+ private final int perRequestPages;
+
+ private int pageUsage;
+
+ public LazyMemorySegmentPool(Object owner, MemoryManager memoryManager, int maxPages) {
+ this(owner, memoryManager, maxPages, false);
+ }
+
+ public LazyMemorySegmentPool(Object owner, MemoryManager memoryManager, int maxPages, boolean freeMore) {
+ this.owner = owner;
+ this.memoryManager = memoryManager;
+ this.freeMore = freeMore;
+ this.cachePages = new ArrayList<>();
+ this.maxPages = maxPages;
+ this.pageUsage = 0;
+ this.perRequestPages = Math.max(1, (int) (PER_REQUEST_MEMORY_SIZE / memoryManager.getPageSize()));
+ }
+
+ @Override
+ public int pageSize() {
+ return this.memoryManager.getPageSize();
+ }
+
+ @Override
+ public void returnAll(List<MemorySegment> memory) {
+ this.pageUsage -= memory.size();
+ if (this.pageUsage < 0) {
+ throw new RuntimeException("Return too more memories.");
+ }
+ this.cachePages.addAll(memory);
+
+ if (this.freeMore && this.cachePages.size() >= this.perRequestPages) {
+ this.memoryManager.release(this.cachePages);
+ }
+ }
+
+ public void returnPage(MemorySegment segment) {
+ returnAll(Collections.singletonList(segment));
+ }
+
+ @Override
+ public MemorySegment nextSegment() {
+ int freePages = freePages();
+ if (freePages == 0) {
+ return null;
+ }
+
+ if (this.cachePages.isEmpty()) {
+ int numPages = Math.min(freePages, this.perRequestPages);
+ try {
+ this.memoryManager.allocatePages(owner, this.cachePages, numPages);
+ } catch (MemoryAllocationException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ this.pageUsage++;
+ return this.cachePages.remove(this.cachePages.size() - 1);
+ }
+
+ @Override
+ public int freePages() {
+ return this.maxPages - this.pageUsage;
+ }
+
+ @Override
+ public void close() {
+ if (this.pageUsage != 0) {
+ throw new RuntimeException(
+ "Should return all used memory before clean, page used: " + pageUsage);
+ }
+ cleanCache();
+ }
+
+ public void cleanCache() {
+ this.memoryManager.release(this.cachePages);
Review comment:
No, this is a manual release of excess memory. This frequency will not be too high.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/143532669 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
Hash:5628de7fba5004f793d14a16fff55186ce7f3d41 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144087946 TriggerType:PUSH TriggerID:5628de7fba5004f793d14a16fff55186ce7f3d41
Hash:5628de7fba5004f793d14a16fff55186ce7f3d41 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277 TriggerType:PUSH TriggerID:5628de7fba5004f793d14a16fff55186ce7f3d41
-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187)
* 5628de7fba5004f793d14a16fff55186ce7f3d41 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144087946) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187",
"triggerID" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"triggerType" : "PUSH"
}, {
"hash" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"status" : "FAILURE",
"url" : "https://travis-ci.com/flink-ci/flink/builds/143532669",
"triggerID" : "e904ab518c4d3b26bf27a89c61b9df30af54b6ed",
"triggerType" : "PUSH"
}, {
"hash" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/flink-ci/flink/builds/144087946",
"triggerID" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"triggerType" : "PUSH"
}, {
"hash" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277",
"triggerID" : "5628de7fba5004f793d14a16fff55186ce7f3d41",
"triggerType" : "PUSH"
}, {
"hash" : "a32248db5be831164281fff2448a346c91645817",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a32248db5be831164281fff2448a346c91645817",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187)
* 5628de7fba5004f793d14a16fff55186ce7f3d41 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144087946) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277)
* a32248db5be831164281fff2448a346c91645817 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] JingsongLi commented on a change in pull request #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on a change in pull request #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#discussion_r365632450
##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/util/LazyMemorySegmentPool.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.util;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.memory.MemoryAllocationException;
+import org.apache.flink.runtime.memory.MemoryManager;
+
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * {@link MemorySegmentPool} that lazy allocate memory pages from {@link MemoryManager}.
+ */
+public class LazyMemorySegmentPool implements MemorySegmentPool, Closeable {
+
+ private static final long PER_REQUEST_MEMORY_SIZE = 16 * 1024 * 1024;
+
+ private final Object owner;
+ private final MemoryManager memoryManager;
+ private final boolean freeMore;
+ private final ArrayList<MemorySegment> cachePages;
+ private final int maxPages;
+ private final int perRequestPages;
+
+ private int pageUsage;
+
+ public LazyMemorySegmentPool(Object owner, MemoryManager memoryManager, int maxPages) {
+ this(owner, memoryManager, maxPages, false);
+ }
+
+ public LazyMemorySegmentPool(Object owner, MemoryManager memoryManager, int maxPages, boolean freeMore) {
+ this.owner = owner;
+ this.memoryManager = memoryManager;
+ this.freeMore = freeMore;
+ this.cachePages = new ArrayList<>();
+ this.maxPages = maxPages;
+ this.pageUsage = 0;
+ this.perRequestPages = Math.max(1, (int) (PER_REQUEST_MEMORY_SIZE / memoryManager.getPageSize()));
+ }
+
+ @Override
+ public int pageSize() {
+ return this.memoryManager.getPageSize();
+ }
+
+ @Override
+ public void returnAll(List<MemorySegment> memory) {
+ this.pageUsage -= memory.size();
+ if (this.pageUsage < 0) {
+ throw new RuntimeException("Return too more memories.");
+ }
+ this.cachePages.addAll(memory);
+
+ if (this.freeMore && this.cachePages.size() >= this.perRequestPages) {
Review comment:
not used now, I'll delete it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TsReaper commented on a change in pull request #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
TsReaper commented on a change in pull request #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#discussion_r365168981
##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/util/LazyMemorySegmentPool.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.util;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.memory.MemoryAllocationException;
+import org.apache.flink.runtime.memory.MemoryManager;
+
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * {@link MemorySegmentPool} that lazy allocate memory pages from {@link MemoryManager}.
+ */
+public class LazyMemorySegmentPool implements MemorySegmentPool, Closeable {
+
+ private static final long PER_REQUEST_MEMORY_SIZE = 16 * 1024 * 1024;
+
+ private final Object owner;
+ private final MemoryManager memoryManager;
+ private final boolean freeMore;
+ private final ArrayList<MemorySegment> cachePages;
+ private final int maxPages;
+ private final int perRequestPages;
+
+ private int pageUsage;
+
+ public LazyMemorySegmentPool(Object owner, MemoryManager memoryManager, int maxPages) {
+ this(owner, memoryManager, maxPages, false);
+ }
+
+ public LazyMemorySegmentPool(Object owner, MemoryManager memoryManager, int maxPages, boolean freeMore) {
+ this.owner = owner;
+ this.memoryManager = memoryManager;
+ this.freeMore = freeMore;
+ this.cachePages = new ArrayList<>();
+ this.maxPages = maxPages;
+ this.pageUsage = 0;
+ this.perRequestPages = Math.max(1, (int) (PER_REQUEST_MEMORY_SIZE / memoryManager.getPageSize()));
+ }
+
+ @Override
+ public int pageSize() {
+ return this.memoryManager.getPageSize();
+ }
+
+ @Override
+ public void returnAll(List<MemorySegment> memory) {
+ this.pageUsage -= memory.size();
+ if (this.pageUsage < 0) {
+ throw new RuntimeException("Return too more memories.");
+ }
+ this.cachePages.addAll(memory);
+
+ if (this.freeMore && this.cachePages.size() >= this.perRequestPages) {
Review comment:
Is this `freeMore` mode actually in use? How much does it impact the performance?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] JingsongLi commented on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-572915567
travis passed in https://travis-ci.org/JingsongLi/flink/builds/634138477
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/143532669 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
Hash:5628de7fba5004f793d14a16fff55186ce7f3d41 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144087946 TriggerType:PUSH TriggerID:5628de7fba5004f793d14a16fff55186ce7f3d41
Hash:5628de7fba5004f793d14a16fff55186ce7f3d41 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277 TriggerType:PUSH TriggerID:5628de7fba5004f793d14a16fff55186ce7f3d41
-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187)
* 5628de7fba5004f793d14a16fff55186ce7f3d41 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144087946) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/143532669 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932
<!--
Meta data
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
Hash:e904ab518c4d3b26bf27a89c61b9df30af54b6ed Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/143532669 TriggerType:PUSH TriggerID:e904ab518c4d3b26bf27a89c61b9df30af54b6ed
-->
## CI report:
* e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] JingsongLi commented on issue #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#issuecomment-590144331
Thanks @TsReaper for the reviewing. Merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TsReaper commented on a change in pull request #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily
allocate memory
Posted by GitBox <gi...@apache.org>.
TsReaper commented on a change in pull request #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
URL: https://github.com/apache/flink/pull/10797#discussion_r365084839
##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/sort/BinaryExternalSorter.java
##########
@@ -1054,7 +1018,7 @@ private void releaseEmptyBuffers() {
}
}
}
- releaseCoreSegments();
+ sortReadMemory.forEach(LazyMemorySegmentPool::cleanCache);
Review comment:
We don't need to manually clean cache, as memory pool will try to clean cache when a segment is returned.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services