You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/04/27 14:00:45 UTC
[GitHub] [flink] spurthicriteo opened a new pull request #11923: [FLINK-17401] Add Labels to the mesos task info
spurthicriteo opened a new pull request #11923:
URL: https://github.com/apache/flink/pull/11923
## What is the purpose of the change
[FLINK-17401] Currently labels are not set in the task info object. A lot of companies can have logic specific to labels on TM. For example in criteo, based on labels we set kerberos env variables and mesos debug capabilities. It is critical for us to be able to pass these label values to the task manager.
## Brief change log
Reading the labels as map structure from flink config and pass it to mesos task parameters. TaskInfo object is populated with the labels by LaunchableMesosWorker.
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: don't know
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? JavaDocs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-637469526
I'm closing this PR now. Please reopen if you want to continue working on it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-620020496
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c1f20e60592bc804a4f6260221c96d564afa50b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-620020496
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/162238046",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
}, {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=330",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c1f20e60592bc804a4f6260221c96d564afa50b Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/162238046) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=330)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-620020496
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/162238046",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
}, {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=330",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
}, {
"hash" : "ffc0831893642d434839fc397fb9e9ec5a4dcad2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ffc0831893642d434839fc397fb9e9ec5a4dcad2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c1f20e60592bc804a4f6260221c96d564afa50b Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/162238046) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=330)
* ffc0831893642d434839fc397fb9e9ec5a4dcad2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-785088304
I see, if you are personally benefiting from this change, then I think it makes sense. Go ahead and open a new PR to address the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-621637411
Thanks a lot for this contribution. Our CI system found one issue with the docs
```
Option mesos.resourcemanager.tasks.labels in class org.apache.flink.mesos.runtime.clusterframework.MesosTaskManagerParameters is not documented.
```
Could you address this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] qinghui-xu commented on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
qinghui-xu commented on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-784532410
Hello, I'd like to continue to work on this based on @spurthicriteo's work. Will submit a new PR soon to address the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-621639318
It would be nice if you could add a test that ensures configured labels are properly forwarded / set in Mesos. If there's no good test in the java codebase, you could consider extending https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/test_mesos_wordcount.sh.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-620020496
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/162238046",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
}, {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=330",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
}, {
"hash" : "ffc0831893642d434839fc397fb9e9ec5a4dcad2",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/162343762",
"triggerID" : "ffc0831893642d434839fc397fb9e9ec5a4dcad2",
"triggerType" : "PUSH"
}, {
"hash" : "ffc0831893642d434839fc397fb9e9ec5a4dcad2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=341",
"triggerID" : "ffc0831893642d434839fc397fb9e9ec5a4dcad2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c1f20e60592bc804a4f6260221c96d564afa50b Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/162238046) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=330)
* ffc0831893642d434839fc397fb9e9ec5a4dcad2 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/162343762) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=341)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-633933320
@spurthicriteo What's the status of this pull request? Please let us know if you do not have the time to work on this PR anymore. We can then decide what to do with it (I tend to close it, but keep the JIRA open for others to pick up your work)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-620020496
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/162238046",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
}, {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=330",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
}, {
"hash" : "ffc0831893642d434839fc397fb9e9ec5a4dcad2",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/162343762",
"triggerID" : "ffc0831893642d434839fc397fb9e9ec5a4dcad2",
"triggerType" : "PUSH"
}, {
"hash" : "ffc0831893642d434839fc397fb9e9ec5a4dcad2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=341",
"triggerID" : "ffc0831893642d434839fc397fb9e9ec5a4dcad2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffc0831893642d434839fc397fb9e9ec5a4dcad2 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/162343762) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=341)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-785014582
Are you using Mesos in production? Or are you interested in contributing to Apache Flink in general?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-620007620
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 0c1f20e60592bc804a4f6260221c96d564afa50b (Mon Apr 27 14:04:01 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-17401).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-620020496
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/162238046",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
}, {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=330",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c1f20e60592bc804a4f6260221c96d564afa50b Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/162238046) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=330)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger closed pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
rmetzger closed pull request #11923:
URL: https://github.com/apache/flink/pull/11923
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-620020496
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/162238046",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
}, {
"hash" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=330",
"triggerID" : "0c1f20e60592bc804a4f6260221c96d564afa50b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c1f20e60592bc804a4f6260221c96d564afa50b Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/162238046) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=330)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger edited a comment on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
rmetzger edited a comment on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-785088304
I see, if you are personally benefiting from this change, then I think it makes sense. Go ahead and open a new PR to address the comments (given that Mesos is a fairly inactive project (even discussing shutting the project down), I didn't want you to put effort into the mesos integration of Flink -- but your case is different)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] qinghui-xu commented on pull request #11923: [FLINK-17401] Add Labels to the mesos task info
Posted by GitBox <gi...@apache.org>.
qinghui-xu commented on pull request #11923:
URL: https://github.com/apache/flink/pull/11923#issuecomment-785060259
Yes, we are using mesos in production in Criteo. And personally I'm interested in the Flink project and would like to make contributions in a more general sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org