You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/02/07 13:46:02 UTC
[GitHub] [flink] azagrebin opened a new pull request #11039: [FLINK-15942]
Do not log huge/infinite cpu/memory resources in profiles
azagrebin opened a new pull request #11039: [FLINK-15942] Do not log huge/infinite cpu/memory resources in profiles
URL: https://github.com/apache/flink/pull/11039
## What is the purpose of the change
In local execution mode, we assume that cpu and task memory of slots are infinite to match any requests. Those resources are configured to `Double/Long.MAX_VALUE` values in this case. These ridiculously huge numbers get logged as slot's `ResourceProfile` and can confuse users. As a quick fix, we do not output the values which exceed a certain huge threshold.
## Brief change log
Adjust `ResourceProfile#toString`.
## Verifying this change
unit tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11039: [FLINK-15942] Do
not log huge/infinite cpu/memory resources in profiles
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11039: [FLINK-15942] Do not log huge/infinite cpu/memory resources in profiles
URL: https://github.com/apache/flink/pull/11039#issuecomment-583410374
<!--
Meta data
Hash:43b4289c0851b93da7d57d97e911f8b8091f11f5 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4957 TriggerType:PUSH TriggerID:43b4289c0851b93da7d57d97e911f8b8091f11f5
Hash:43b4289c0851b93da7d57d97e911f8b8091f11f5 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147928742 TriggerType:PUSH TriggerID:43b4289c0851b93da7d57d97e911f8b8091f11f5
-->
## CI report:
* 43b4289c0851b93da7d57d97e911f8b8091f11f5 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147928742) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4957)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11039: [FLINK-15942] Do not
log huge/infinite cpu/memory resources in profiles
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11039: [FLINK-15942] Do not log huge/infinite cpu/memory resources in profiles
URL: https://github.com/apache/flink/pull/11039#issuecomment-583410374
<!--
Meta data
Hash:43b4289c0851b93da7d57d97e911f8b8091f11f5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:43b4289c0851b93da7d57d97e911f8b8091f11f5
-->
## CI report:
* 43b4289c0851b93da7d57d97e911f8b8091f11f5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #11039:
[FLINK-15942] Do not log huge/infinite cpu/memory resources in profiles
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #11039: [FLINK-15942] Do not log huge/infinite cpu/memory resources in profiles
URL: https://github.com/apache/flink/pull/11039#discussion_r376501880
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/clusterframework/types/ResourceProfileTest.java
##########
@@ -387,4 +392,36 @@ public void testSingletonPropertyOfAny() throws Exception {
assertSame(ResourceProfile.ANY, copiedProfile);
}
+
+ @Test
+ public void testToStringWithoutCpu() {
+ double notLoggedCpu = MAX_CPU_CORE_NUMBER_TO_LOG.doubleValue() + 1.0;
+ ResourceProfile resourceProfile = getResourceProfileForLogsWithCpuVal(notLoggedCpu);
+ assertThat(resourceProfile.toString(), is(getExpectedToString(resourceProfile, "")));
+ }
+
+ @Test
+ public void testToStringWithCpu() {
+ ResourceProfile resourceProfile = getResourceProfileForLogsWithCpuVal(1.0);
+ assertThat(resourceProfile.toString(), is(getExpectedToString(resourceProfile, "cpuCores=1.0, ")));
Review comment:
I think we could simplify the assertion a bit by testing whether the string contains `cpuCores` and a configured memory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #11039:
[FLINK-15942] Do not log huge/infinite cpu/memory resources in profiles
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #11039: [FLINK-15942] Do not log huge/infinite cpu/memory resources in profiles
URL: https://github.com/apache/flink/pull/11039#discussion_r376503812
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/clusterframework/types/ResourceProfileTest.java
##########
@@ -387,4 +392,36 @@ public void testSingletonPropertyOfAny() throws Exception {
assertSame(ResourceProfile.ANY, copiedProfile);
}
+
+ @Test
+ public void testToStringWithoutCpu() {
+ double notLoggedCpu = MAX_CPU_CORE_NUMBER_TO_LOG.doubleValue() + 1.0;
+ ResourceProfile resourceProfile = getResourceProfileForLogsWithCpuVal(notLoggedCpu);
+ assertThat(resourceProfile.toString(), is(getExpectedToString(resourceProfile, "")));
+ }
+
+ @Test
+ public void testToStringWithCpu() {
+ ResourceProfile resourceProfile = getResourceProfileForLogsWithCpuVal(1.0);
+ assertThat(resourceProfile.toString(), is(getExpectedToString(resourceProfile, "cpuCores=1.0, ")));
Review comment:
Alright, I would just make sure that the right positions of commas are covered
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann closed pull request #11039: [FLINK-15942] Do
not log huge/infinite cpu/memory resources in profiles
Posted by GitBox <gi...@apache.org>.
tillrohrmann closed pull request #11039: [FLINK-15942] Do not log huge/infinite cpu/memory resources in profiles
URL: https://github.com/apache/flink/pull/11039
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #11039:
[FLINK-15942] Do not log huge/infinite cpu/memory resources in profiles
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #11039: [FLINK-15942] Do not log huge/infinite cpu/memory resources in profiles
URL: https://github.com/apache/flink/pull/11039#discussion_r376491735
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/clusterframework/types/ResourceProfileTest.java
##########
@@ -29,17 +29,22 @@
import java.util.ArrayList;
import java.util.List;
+import static org.apache.flink.runtime.clusterframework.types.ResourceProfile.MAX_CPU_CORE_NUMBER_TO_LOG;
+import static org.apache.flink.runtime.clusterframework.types.ResourceProfile.MAX_MEMORY_SIZE_TO_LOG;
+import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
/**
* Tests for the {@link ResourceProfile}.
*/
public class ResourceProfileTest {
Review comment:
`extends TestLogger` is missing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11039: [FLINK-15942] Do not
log huge/infinite cpu/memory resources in profiles
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11039: [FLINK-15942] Do not log huge/infinite cpu/memory resources in profiles
URL: https://github.com/apache/flink/pull/11039#issuecomment-583396287
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 43b4289c0851b93da7d57d97e911f8b8091f11f5 (Fri Feb 07 13:48:36 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11039: [FLINK-15942] Do
not log huge/infinite cpu/memory resources in profiles
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11039: [FLINK-15942] Do not log huge/infinite cpu/memory resources in profiles
URL: https://github.com/apache/flink/pull/11039#issuecomment-583410374
<!--
Meta data
Hash:43b4289c0851b93da7d57d97e911f8b8091f11f5 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4957 TriggerType:PUSH TriggerID:43b4289c0851b93da7d57d97e911f8b8091f11f5
Hash:43b4289c0851b93da7d57d97e911f8b8091f11f5 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147928742 TriggerType:PUSH TriggerID:43b4289c0851b93da7d57d97e911f8b8091f11f5
-->
## CI report:
* 43b4289c0851b93da7d57d97e911f8b8091f11f5 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147928742) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4957)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services