You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Hadoop QA (Jira)" <ji...@apache.org> on 2021/01/25 23:25:00 UTC
[jira] [Commented] (TEZ-3984) Shuffle: Out of Band DME event
sending causes errors
[ https://issues.apache.org/jira/browse/TEZ-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271724#comment-17271724 ]
Hadoop QA commented on TEZ-3984:
--------------------------------
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 49s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} |
|| || || || {color:brown} branch-0.9 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 8s{color} | {color:green} branch-0.9 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} branch-0.9 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 16s{color} | {color:green} branch-0.9 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} branch-0.9 passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 35s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 32s{color} | {color:green} branch-0.9 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 18s{color} | {color:orange} tez-runtime-library: The patch generated 1 new + 241 unchanged - 0 fixed = 242 total (was 241) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 29s{color} | {color:green} tez-runtime-library in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 30s{color} | {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/91/artifact/out/Dockerfile |
| JIRA Issue | TEZ-3984 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12939327/TEZ-3984.2-branch-0.9.patch |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 200d7de5c6c5 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | branch-0.9 / d6978fe16 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/91/artifact/out/diff-checkstyle-tez-runtime-library.txt |
| Test Results | https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/91/testReport/ |
| Max. process+thread count | 104 (vs. ulimit of 5500) |
| modules | C: tez-runtime-library U: tez-runtime-library |
| Console output | https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/91/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Shuffle: Out of Band DME event sending causes errors
> ----------------------------------------------------
>
> Key: TEZ-3984
> URL: https://issues.apache.org/jira/browse/TEZ-3984
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.8.4, 0.9.1, 0.10.0
> Reporter: Gopal Vijayaraghavan
> Assignee: Jaume M
> Priority: Critical
> Labels: correctness
> Fix For: 0.10.1
>
> Attachments: TEZ-3984-branch-0.9.patch, TEZ-3984.1.patch, TEZ-3984.2-branch-0.9.patch, TEZ-3984.2.patch, TEZ-3984.3.patch, TEZ-3984.4.patch, TEZ-3984.5.patch, TEZ-3984.5.patch
>
>
> In case of a task Input throwing an exception, the outputs are also closed in the LogicalIOProcessorRuntimeTask.cleanup().
> Cleanup ignore all the events returned by output close, however if any output tries to send an event out of band by directly calling outputContext.sendEvents(events), then those events can reach the AM before the task failure is reported.
> This can cause correctness issues with shuffle since zero sized events can be sent out due to an input failure and downstream tasks may never reattempt a fetch from the valid attempt.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)