You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/08/27 16:13:53 UTC
[GitHub] [flink] azagrebin opened a new pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
azagrebin opened a new pull request #13265:
URL: https://github.com/apache/flink/pull/13265
#12980 increased the waiting time to GC all allocated memory to verify that nothing is allocated in `MemoryManager`/`UnsafeMemoryBudget`. The waiting happens in a separate thread in production and does not disrupt anything. The problem is that tests use the same method which blocks the testing thread for too long. We do not need such big waiting time in tests. The PR refactors code to get back to the smaller waiting time in tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13265:
URL: https://github.com/apache/flink/pull/13265#issuecomment-682048215
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit af69156ea6dbc7513cb7dde023cdf8cc454bec31 (Thu Aug 27 16:15:25 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #13265:
URL: https://github.com/apache/flink/pull/13265#discussion_r478863140
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java
##########
@@ -36,16 +36,20 @@
*/
class UnsafeMemoryBudget {
private static final int MAX_SLEEPS = 11; // 2^11 - 1 = (2 x 1024) - 1 ms ~ 2 s total sleep duration
- private static final int MAX_SLEEPS_VERIFY_EMPTY = 17; // 2^17 - 1 = (128 x 1024) - 1 ms ~ 2 min total sleep duration
+ static final int MAX_SLEEPS_VERIFY_EMPTY_FOR_TESTS = 10; // 2^10 - 1 = (1 x 1024) - 1 ms ~ 1 s total sleep duration
Review comment:
Testing code should not be contained in production code.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java
##########
@@ -89,12 +89,14 @@
*
* @param memorySize The total size of the off-heap memory to be managed by this memory manager.
* @param pageSize The size of the pages handed out by the memory manager.
+ * @param verifyEmptyWaitGcMaxSleeps defines how long to wait for GC of all allocated memory to check for memory leaks,
+ * see also {@link UnsafeMemoryBudget}.
Review comment:
Hmm, the description says that it is the time to wait for GC but isn't is the number of exponential retries?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13265:
URL: https://github.com/apache/flink/pull/13265#issuecomment-682059106
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af69156ea6dbc7513cb7dde023cdf8cc454bec31",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5936",
"triggerID" : "af69156ea6dbc7513cb7dde023cdf8cc454bec31",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af69156ea6dbc7513cb7dde023cdf8cc454bec31 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5936)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13265:
URL: https://github.com/apache/flink/pull/13265#issuecomment-682059106
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af69156ea6dbc7513cb7dde023cdf8cc454bec31",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5936",
"triggerID" : "af69156ea6dbc7513cb7dde023cdf8cc454bec31",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af69156ea6dbc7513cb7dde023cdf8cc454bec31 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5936)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13265:
URL: https://github.com/apache/flink/pull/13265#issuecomment-682059106
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af69156ea6dbc7513cb7dde023cdf8cc454bec31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5936",
"triggerID" : "af69156ea6dbc7513cb7dde023cdf8cc454bec31",
"triggerType" : "PUSH"
}, {
"hash" : "92e66f032e705f7ce95b4bcb68c494f4dac0529a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5977",
"triggerID" : "92e66f032e705f7ce95b4bcb68c494f4dac0529a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 92e66f032e705f7ce95b4bcb68c494f4dac0529a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5977)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #13265:
URL: https://github.com/apache/flink/pull/13265#discussion_r478886087
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java
##########
@@ -89,12 +89,14 @@
*
* @param memorySize The total size of the off-heap memory to be managed by this memory manager.
* @param pageSize The size of the pages handed out by the memory manager.
+ * @param verifyEmptyWaitGcMaxSleeps defines how long to wait for GC of all allocated memory to check for memory leaks,
+ * see also {@link UnsafeMemoryBudget}.
Review comment:
You are right, I just did not want to repeat all details here, so I referred to `UnsafeMemoryBudget` for full description. At the end, the number of retries translates into the waiting time anyways.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13265:
URL: https://github.com/apache/flink/pull/13265#issuecomment-682059106
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af69156ea6dbc7513cb7dde023cdf8cc454bec31",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5936",
"triggerID" : "af69156ea6dbc7513cb7dde023cdf8cc454bec31",
"triggerType" : "PUSH"
}, {
"hash" : "92e66f032e705f7ce95b4bcb68c494f4dac0529a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92e66f032e705f7ce95b4bcb68c494f4dac0529a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af69156ea6dbc7513cb7dde023cdf8cc454bec31 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5936)
* 92e66f032e705f7ce95b4bcb68c494f4dac0529a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #13265:
URL: https://github.com/apache/flink/pull/13265#discussion_r478885574
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java
##########
@@ -36,16 +36,20 @@
*/
class UnsafeMemoryBudget {
private static final int MAX_SLEEPS = 11; // 2^11 - 1 = (2 x 1024) - 1 ms ~ 2 s total sleep duration
- private static final int MAX_SLEEPS_VERIFY_EMPTY = 17; // 2^17 - 1 = (128 x 1024) - 1 ms ~ 2 min total sleep duration
+ static final int MAX_SLEEPS_VERIFY_EMPTY_FOR_TESTS = 10; // 2^10 - 1 = (1 x 1024) - 1 ms ~ 1 s total sleep duration
Review comment:
True, I can move to the testing builder
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13265:
URL: https://github.com/apache/flink/pull/13265#issuecomment-682059106
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af69156ea6dbc7513cb7dde023cdf8cc454bec31",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5936",
"triggerID" : "af69156ea6dbc7513cb7dde023cdf8cc454bec31",
"triggerType" : "PUSH"
}, {
"hash" : "92e66f032e705f7ce95b4bcb68c494f4dac0529a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5977",
"triggerID" : "92e66f032e705f7ce95b4bcb68c494f4dac0529a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af69156ea6dbc7513cb7dde023cdf8cc454bec31 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5936)
* 92e66f032e705f7ce95b4bcb68c494f4dac0529a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5977)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #13265:
URL: https://github.com/apache/flink/pull/13265#discussion_r478886087
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java
##########
@@ -89,12 +89,14 @@
*
* @param memorySize The total size of the off-heap memory to be managed by this memory manager.
* @param pageSize The size of the pages handed out by the memory manager.
+ * @param verifyEmptyWaitGcMaxSleeps defines how long to wait for GC of all allocated memory to check for memory leaks,
+ * see also {@link UnsafeMemoryBudget}.
Review comment:
You are right, I just did not want to repeat all details here, so I referred to `UnsafeMemoryBudget` for full description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin closed pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
Posted by GitBox <gi...@apache.org>.
azagrebin closed pull request #13265:
URL: https://github.com/apache/flink/pull/13265
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13265: [FLINK-19055] Wait less time for all memory GC in tests (MemoryManager#verifyEmpty)
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13265:
URL: https://github.com/apache/flink/pull/13265#issuecomment-682059106
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af69156ea6dbc7513cb7dde023cdf8cc454bec31",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af69156ea6dbc7513cb7dde023cdf8cc454bec31",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af69156ea6dbc7513cb7dde023cdf8cc454bec31 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org