You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Tez CI (Jira)" <ji...@apache.org> on 2021/02/16 04:11:00 UTC
[jira] [Commented] (TEZ-4192) Attach task specific info in
scheduler and retrieve in communicator
[ https://issues.apache.org/jira/browse/TEZ-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17285021#comment-17285021 ]
Tez CI commented on TEZ-4192:
-----------------------------
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 9 new or modified test files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 4m 57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 46s{color} | {color:green} master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s{color} | {color:green} master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s{color} | {color:green} master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s{color} | {color:green} master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 45s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 26s{color} | {color:orange} tez-dag: The patch generated 12 new + 285 unchanged - 5 fixed = 297 total (was 290) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 9s{color} | {color:orange} tez-ext-service-tests: The patch generated 2 new + 9 unchanged - 0 fixed = 11 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 55s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 14s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 57s{color} | {color:green} tez-ext-service-tests in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 50s{color} | {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-69/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/69 |
| JIRA Issue | TEZ-4192 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux c5a2270ff96f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 3089d478f |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-69/1/artifact/out/diff-checkstyle-tez-dag.txt |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-69/1/artifact/out/diff-checkstyle-tez-ext-service-tests.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-69/1/testReport/ |
| Max. process+thread count | 907 (vs. ulimit of 5500) |
| modules | C: tez-api tez-dag tez-ext-service-tests U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-69/1/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Attach task specific info in scheduler and retrieve in communicator
> -------------------------------------------------------------------
>
> Key: TEZ-4192
> URL: https://issues.apache.org/jira/browse/TEZ-4192
> Project: Apache Tez
> Issue Type: Improvement
> Affects Versions: 0.9.2
> Reporter: Mustafa İman
> Assignee: Mustafa İman
> Priority: Major
> Attachments: TEZ-4192.1.patch, TEZ-4192.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is no way to pass information from scheduler to communicator. Sometimes we want to let communicator know about extra properties of the task other than its spec. One example is if a task is preemptable.
> When scheduler receives a task, its taskspec is already constructed. It is a nice immutable object at this point. We do not want to modify and put more info in it at that stage. So we pass a new piece of information to TaskSchedulerContext#taskAllocated along with the allocation. This info is passed to the communicator untouched. It is up to the communicator implementation to make use of it. Communicator implementation needs to override a new registerRunningTaskAttempt that accepts this additional object along with all the other arguments in the original registerRunningTaskAttempt method.
> The new registerRunningTaskAttempt method has a default implementation that falls back to the old method. So if the user does not override the new method, the same old behavior is preserved. Similarly on scheduler side, the user needs to call new taskAllocated method to pass the additional info object. If they use the old taskAllocated method, old behavior is preserved.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)