You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "TezQA (Jira)" <ji...@apache.org> on 2020/05/14 16:40:00 UTC
[jira] [Commented] (TEZ-4181) [Kubernetes] Use hostname + pod UID
for shuffle manager caching
[ https://issues.apache.org/jira/browse/TEZ-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107456#comment-17107456 ]
TezQA commented on TEZ-4181:
----------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} prototool {color} | {color:blue} 0m 0s{color} | {color:blue} prototool was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 7s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 13s{color} | {color:orange} tez-api: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 18s{color} | {color:orange} tez-runtime-library: The patch generated 25 new + 134 unchanged - 0 fixed = 159 total (was 134) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 22s{color} | {color:red} tez-runtime-library generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s{color} | {color:green} tez-runtime-internals in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 29s{color} | {color:green} tez-runtime-library in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 37s{color} | {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:tez-runtime-library |
| | org.apache.tez.runtime.library.common.shuffle.HostPort is incompatible with expected argument type org.apache.tez.runtime.library.common.shuffle.HostInfo in org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager$FetchFutureCallback.onSuccess(FetchResult) At ShuffleManager.java:argument type org.apache.tez.runtime.library.common.shuffle.HostInfo in org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager$FetchFutureCallback.onSuccess(FetchResult) At ShuffleManager.java:[line 1169] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/PreCommit-TEZ-Build/431/artifact/out/Dockerfile |
| JIRA Issue | TEZ-4181 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13002963/TEZ-4181.patch |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile cc prototool |
| uname | Linux 39752cf93b13 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 354c2a4 |
| Default Java | 1.8.0_252 |
| checkstyle | https://builds.apache.org/job/PreCommit-TEZ-Build/431/artifact/out/diff-checkstyle-tez-api.txt |
| checkstyle | https://builds.apache.org/job/PreCommit-TEZ-Build/431/artifact/out/diff-checkstyle-tez-runtime-library.txt |
| findbugs | https://builds.apache.org/job/PreCommit-TEZ-Build/431/artifact/out/new-findbugs-tez-runtime-library.html |
| Test Results | https://builds.apache.org/job/PreCommit-TEZ-Build/431/testReport/ |
| Max. process+thread count | 242 (vs. ulimit of 5500) |
| modules | C: tez-api tez-runtime-internals tez-runtime-library U: . |
| Console output | https://builds.apache.org/job/PreCommit-TEZ-Build/431/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> [Kubernetes] Use hostname + pod UID for shuffle manager caching
> ---------------------------------------------------------------
>
> Key: TEZ-4181
> URL: https://issues.apache.org/jira/browse/TEZ-4181
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Attila Magyar
> Assignee: Attila Magyar
> Priority: Major
> Attachments: TEZ-4181.patch
>
>
> When a pod restarts, it uses the same hostname and shuffle port. Now when fetcher threads connects to download the shuffle data it will use the cached connection info and since the pod has died it's shuffle data will also get cleaned up. When the pod restarts, it receives connection from clients to download specific shuffle data but the daemon will not have it because of the restart.
> In ShuffleManager.java's knownSrcHosts the key should be updated to HostInfo which is a combination of host+port and the host's unique ID. The host host Id changes when a node is killed or restarted.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)