You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by GitBox <gi...@apache.org> on 2021/04/26 09:15:22 UTC
[GitHub] [tez] kishendas opened a new pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
kishendas opened a new pull request #120:
URL: https://github.com/apache/tez/pull/120
Following exception is thrown whenever we do ProtoMessageWriter.hflush on S3, which internally calls S3ABlockOutputStream.hflush which is not supported and it throws java.lang.UnsupportedOperationException.
bdffe22d96ae [mdc@18060 class="yarn.YarnUncaughtExceptionHandler" level="ERROR" thread="HistoryEventHandlingThread"] Thread Thread[HistoryEventHandlingThread, 5,main] threw an Exception.^Mjava.lang.UnsupportedOperationException: S3A streams are not Syncable^M at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.hflush(S3ABlockOutputStream.java:657)^M at org.apache.hadoop.fs.FS DataOutputStream.hflush(FSDataOutputStream.java:136)^M at org.apache.hadoop.io.SequenceFile$Writer.hflush(SequenceFile.java:1367)^M at org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(ProtoMessageWr iter.java:64)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.finishCurrentDag(ProtoHistoryLoggingService.java:239)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.han dleEvent(ProtoHistoryLoggingService.java:198)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.loop(ProtoHistoryLoggingService.java:153)^M at java.lang.Thread.r
un(Thread.java:748)^M
In order to fix this issue, we should first check if the method is supported or not using StreamCapabilities, before invoking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-841627959
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 4m 30s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 8m 29s | master passed |
| +1 :green_heart: | compile | 1m 38s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 28s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 41s | master passed |
| +1 :green_heart: | javadoc | 1m 40s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 41s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 53s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 53s | the patch passed |
| +1 :green_heart: | compile | 0m 57s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 57s | the patch passed |
| +1 :green_heart: | compile | 0m 50s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 50s | the patch passed |
| -0 :warning: | checkstyle | 0m 9s | tez-common: The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0) |
| -0 :warning: | checkstyle | 0m 8s | tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 47s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 45s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | findbugs | 2m 12s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 30s | tez-common in the patch passed. |
| +1 :green_heart: | unit | 4m 17s | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 0m 21s | tez-protobuf-history-plugin in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 38m 28s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/120 |
| JIRA Issue | TEZ-4305 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 24744d7bff6d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 01847814d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/6/artifact/out/diff-checkstyle-tez-common.txt |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/6/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/6/testReport/ |
| Max. process+thread count | 235 (vs. ulimit of 5500) |
| modules | C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/6/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] kishendas commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
kishendas commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-842538544
@abstractdog I have added the log message. Please commit the PR, if you don't have any other feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
abstractdog commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-843055802
> @abstractdog My bad. Got tricked by my own comment. Fixed it. Good catch.
thanks!
one more this, I can still see checkstyle problems, I think you're using 4 spaces instead of 2 spaces which is used in tez project
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
abstractdog commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-842894619
> @abstractdog I have added the log message. Please commit the PR, if you don't have any other feedback.
I think the log message should go into an else branch...currently, it's logged every time when StreamCapabilities interface is implemented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-839227309
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 5m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 8m 5s | master passed |
| +1 :green_heart: | compile | 2m 34s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 2m 25s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 35s | master passed |
| +1 :green_heart: | javadoc | 2m 33s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 2s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 5m 34s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +0 :ok: | findbugs | 5m 29s | root in master has 2 extant findbugs warnings. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 51s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 4m 3s | root in the patch failed. |
| -1 :x: | compile | 2m 9s | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 2m 9s | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 1m 56s | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 1m 56s | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | checkstyle | 1m 13s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 30s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 9s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | findbugs | 0m 52s | tez-plugins/tez-protobuf-history-plugin generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| -1 :x: | findbugs | 5m 2s | root in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 45s | tez-protobuf-history-plugin in the patch passed. |
| -1 :x: | unit | 3m 47s | root in the patch failed. |
| +1 :green_heart: | asflicense | 0m 55s | The patch does not generate ASF License warnings. |
| | | 61m 53s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:tez-plugins/tez-protobuf-history-plugin |
| | instanceof will always return true for all non-null values in org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(), since all org.apache.hadoop.io.SequenceFile$Writer are instances of org.apache.hadoop.fs.StreamCapabilities At ProtoMessageWriter.java:for all non-null values in org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(), since all org.apache.hadoop.io.SequenceFile$Writer are instances of org.apache.hadoop.fs.StreamCapabilities At ProtoMessageWriter.java:[line 65] |
| Failed junit tests | tez.dag.api.client.rpc.TestDAGClient |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/120 |
| JIRA Issue | TEZ-4305 |
| Optional Tests | dupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle |
| uname | Linux 97d8c24e3c9d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 01847814d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| mvninstall | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt |
| javac | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt |
| compile | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt |
| javac | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt |
| findbugs | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/new-findbugs-tez-plugins_tez-protobuf-history-plugin.html |
| findbugs | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-findbugs-root.txt |
| unit | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-unit-root.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/testReport/ |
| Max. process+thread count | 444 (vs. ulimit of 5500) |
| modules | C: tez-plugins/tez-protobuf-history-plugin . U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog edited a comment on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
abstractdog edited a comment on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-841601704
LGTM +1, just two minor asks:
1. could you please squash the commits for easier later reference?
2. please address checkstyle/whitespace warnings:
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/whitespace-eol.txt
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/diff-checkstyle-root.txt
created TEZ-4307 about Hadoop 3.4.0 to track the fact that this cannot be pushed until we're not upgraded to Hadoop 3.4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog edited a comment on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
abstractdog edited a comment on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-841601704
LGTM +1
could you please squash the commits for easier later reference?
created TEZ-4307 about Hadoop 3.4.0 to track the fact that this cannot be pushed until we're not upgraded to Hadoop 3.4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-843447793
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 13m 40s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 4m 16s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 8m 58s | master passed |
| +1 :green_heart: | compile | 1m 42s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 32s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 40s | master passed |
| +1 :green_heart: | javadoc | 1m 41s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 43s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 55s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 54s | the patch passed |
| +1 :green_heart: | compile | 0m 57s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 57s | the patch passed |
| +1 :green_heart: | compile | 0m 49s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 49s | the patch passed |
| -0 :warning: | checkstyle | 0m 8s | tez-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 49s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 45s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | findbugs | 2m 13s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 31s | tez-common in the patch passed. |
| +1 :green_heart: | unit | 4m 15s | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 0m 22s | tez-protobuf-history-plugin in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 52m 7s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/120 |
| JIRA Issue | TEZ-4305 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 2acb5d3a8b4a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 01847814d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/11/artifact/out/diff-checkstyle-tez-common.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/11/testReport/ |
| Max. process+thread count | 188 (vs. ulimit of 5500) |
| modules | C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/11/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-841498771
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 12m 55s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 5m 19s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 8m 31s | master passed |
| +1 :green_heart: | compile | 4m 6s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 3m 47s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 2m 35s | master passed |
| +1 :green_heart: | javadoc | 3m 52s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 3s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 6m 16s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +0 :ok: | findbugs | 6m 11s | root in master has 2 extant findbugs warnings. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 50s | the patch passed |
| +1 :green_heart: | compile | 3m 58s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 3m 58s | the patch passed |
| +1 :green_heart: | compile | 3m 39s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 3m 39s | the patch passed |
| -0 :warning: | checkstyle | 0m 25s | tez-common: The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0) |
| -0 :warning: | checkstyle | 0m 26s | tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| -0 :warning: | checkstyle | 0m 49s | root: The patch generated 11 new + 1 unchanged - 0 fixed = 12 total (was 1) |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 3m 52s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 20s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | findbugs | 9m 41s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 48s | tez-common in the patch passed. |
| +1 :green_heart: | unit | 4m 32s | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 0m 40s | tez-protobuf-history-plugin in the patch passed. |
| +1 :green_heart: | unit | 76m 3s | root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 56s | The patch does not generate ASF License warnings. |
| | | 171m 33s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/120 |
| JIRA Issue | TEZ-4305 |
| Optional Tests | dupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle |
| uname | Linux 2d01e63a2bd9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 01847814d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/diff-checkstyle-tez-common.txt |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/whitespace-eol.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/testReport/ |
| Max. process+thread count | 1446 (vs. ulimit of 5500) |
| modules | C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin . U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
abstractdog commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-842181947
> @abstractdog Please take a look. We can land this change without waiting for Hadoop 3.4.0 upgrade, as this change will work with both Hadoop 3.4.0 and previous versions. It's just that behavior will be different based on the Hadoop version we use with Tez.
I agree, the latest version is even better because StreamCapabilites is already present in older Hadoop versions
once tez will depend on a hadoop version where HADOOP-17657 present (so writer implements StreamCapabilities), it will behave accordingly
nit: I was wondering if we can place a DEBUG message if hflush is skipped because it's not supported
this codepath is only called from finishCurrentDag, so once per dag, it won't spam the logs...
other than this nit, the patch looks good to me
cc: @harishjp if you have any comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-839016913
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 12m 57s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 5m 3s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 8m 28s | master passed |
| +1 :green_heart: | compile | 2m 32s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 2m 25s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 28s | master passed |
| +1 :green_heart: | javadoc | 2m 30s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 1s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 7m 17s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +0 :ok: | findbugs | 7m 12s | root in master has 2 extant findbugs warnings. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 56s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 5m 38s | root in the patch failed. |
| -1 :x: | compile | 2m 51s | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 2m 51s | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 2m 29s | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 2m 29s | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | checkstyle | 1m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 3m 1s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 23s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | findbugs | 6m 50s | root in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 45s | tez-protobuf-history-plugin in the patch passed. |
| -1 :x: | unit | 4m 48s | root in the patch failed. |
| +1 :green_heart: | asflicense | 0m 49s | The patch does not generate ASF License warnings. |
| | | 82m 48s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | tez.dag.api.client.rpc.TestDAGClient |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/120 |
| JIRA Issue | TEZ-4305 |
| Optional Tests | dupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle |
| uname | Linux 4bc6b059830c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 01847814d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| mvninstall | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt |
| javac | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt |
| compile | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt |
| javac | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt |
| findbugs | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-findbugs-root.txt |
| unit | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-unit-root.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/testReport/ |
| Max. process+thread count | 404 (vs. ulimit of 5500) |
| modules | C: tez-plugins/tez-protobuf-history-plugin . U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] kishendas commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
kishendas commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-840117031
Created a ticket for the unit test failure, as it seems to be unrelated -> https://issues.apache.org/jira/browse/TEZ-4306 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog edited a comment on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
abstractdog edited a comment on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-843055802
> @abstractdog My bad. Got tricked by my own comment. Fixed it. Good catch.
thanks!
one more thing, I can still see checkstyle problems, I think you're using 4 spaces instead of 2 spaces which is used in tez project
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-842934012
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 4m 27s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 8m 52s | master passed |
| +1 :green_heart: | compile | 1m 39s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 30s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 34s | master passed |
| +1 :green_heart: | javadoc | 1m 38s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 17s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 39s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 51s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 58s | the patch passed |
| +1 :green_heart: | compile | 1m 1s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 1s | the patch passed |
| +1 :green_heart: | compile | 0m 54s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 54s | the patch passed |
| -0 :warning: | checkstyle | 0m 10s | tez-common: The patch generated 13 new + 0 unchanged - 0 fixed = 13 total (was 0) |
| -0 :warning: | checkstyle | 0m 8s | tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 51s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | findbugs | 2m 24s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 31s | tez-common in the patch passed. |
| +1 :green_heart: | unit | 4m 47s | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 0m 22s | tez-protobuf-history-plugin in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 39m 37s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/120 |
| JIRA Issue | TEZ-4305 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 692d0099bc8e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 01847814d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/9/artifact/out/diff-checkstyle-tez-common.txt |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/9/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/9/testReport/ |
| Max. process+thread count | 195 (vs. ulimit of 5500) |
| modules | C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/9/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] kishendas commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
kishendas commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-839176316
> @kishendas, we will have to rethink this change to not break users who wish to run against other versions of Hadoop. @abstractdog, we may need to consider separate branch for Hadoop 3.3. There are several API changes and dependencies that are making supporting all of Hadoop 3.x difficult
Please check now. It should now work with older versions of Hadoop as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog merged pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
abstractdog merged pull request #120:
URL: https://github.com/apache/tez/pull/120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-842560012
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 4m 20s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 8m 38s | master passed |
| +1 :green_heart: | compile | 1m 39s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 31s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 41s | master passed |
| +1 :green_heart: | javadoc | 1m 39s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 43s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 52s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 52s | the patch passed |
| +1 :green_heart: | compile | 0m 56s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 56s | the patch passed |
| +1 :green_heart: | compile | 0m 51s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 51s | the patch passed |
| -0 :warning: | checkstyle | 0m 8s | tez-common: The patch generated 12 new + 0 unchanged - 0 fixed = 12 total (was 0) |
| -0 :warning: | checkstyle | 0m 9s | tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 47s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 44s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | findbugs | 2m 13s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 31s | tez-common in the patch passed. |
| +1 :green_heart: | unit | 4m 15s | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 0m 22s | tez-protobuf-history-plugin in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 38m 28s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/120 |
| JIRA Issue | TEZ-4305 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux ad740569d412 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 01847814d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/7/artifact/out/diff-checkstyle-tez-common.txt |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/7/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/7/testReport/ |
| Max. process+thread count | 207 (vs. ulimit of 5500) |
| modules | C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/7/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
abstractdog commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-841601704
LGTM +1
could you please squash you commits for easier later reference?
created TEZ-4307 about Hadoop 3.4.0 to track the fact that this cannot be pushed while we're not upgraded to Hadoop 3.4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] kishendas commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
kishendas commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-842908525
@abstractdog My bad. Got tricked by my own comment. Fixed it. Good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-843614241
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 4m 16s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 8m 35s | master passed |
| +1 :green_heart: | compile | 1m 38s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 31s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 42s | master passed |
| +1 :green_heart: | javadoc | 1m 40s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 42s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 55s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 54s | the patch passed |
| +1 :green_heart: | compile | 0m 56s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 56s | the patch passed |
| +1 :green_heart: | compile | 0m 50s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 50s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 34s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 45s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | findbugs | 2m 15s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 31s | tez-common in the patch passed. |
| +1 :green_heart: | unit | 4m 41s | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 0m 23s | tez-protobuf-history-plugin in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 38m 54s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/120 |
| JIRA Issue | TEZ-4305 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 0c7fa85f8ca3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 867e4d205 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/12/testReport/ |
| Max. process+thread count | 188 (vs. ulimit of 5500) |
| modules | C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/12/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-826688401
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 12m 47s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 13m 2s | master passed |
| +1 :green_heart: | compile | 0m 30s | master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 29s | master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 4s | master passed |
| +1 :green_heart: | javadoc | 0m 37s | master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 26s | master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +0 :ok: | spotbugs | 1m 8s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 7s | master passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 13s | tez-protobuf-history-plugin in the patch failed. |
| -1 :x: | compile | 0m 14s | tez-protobuf-history-plugin in the patch failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javac | 0m 14s | tez-protobuf-history-plugin in the patch failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | compile | 0m 14s | tez-protobuf-history-plugin in the patch failed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08. |
| -1 :x: | javac | 0m 14s | tez-protobuf-history-plugin in the patch failed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08. |
| +1 :green_heart: | checkstyle | 0m 9s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 13s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 13s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| -1 :x: | findbugs | 0m 14s | tez-protobuf-history-plugin in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 14s | tez-protobuf-history-plugin in the patch failed. |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 32m 46s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/120 |
| JIRA Issue | TEZ-4305 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux fa39febd9696 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 01847814d |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| mvninstall | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-mvninstall-tez-plugins_tez-protobuf-history-plugin.txt |
| compile | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-compile-tez-plugins_tez-protobuf-history-plugin-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt |
| javac | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-compile-tez-plugins_tez-protobuf-history-plugin-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt |
| compile | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-compile-tez-plugins_tez-protobuf-history-plugin-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt |
| javac | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-compile-tez-plugins_tez-protobuf-history-plugin-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt |
| findbugs | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-findbugs-tez-plugins_tez-protobuf-history-plugin.txt |
| unit | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-unit-tez-plugins_tez-protobuf-history-plugin.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/testReport/ |
| Max. process+thread count | 96 (vs. ulimit of 5500) |
| modules | C: tez-plugins/tez-protobuf-history-plugin U: tez-plugins/tez-protobuf-history-plugin |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] harishjp commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
harishjp commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-842251967
> nit: I was wondering if we can place a DEBUG message if hflush is skipped because it's not supported
> this codepath is only called from finishCurrentDag, so once per dag, it won't spam the logs...
> other than this nit, the patch looks good to me
> cc: @harishjp if you have any comments
This is fine in tez, we use the same code in hive (It has to be copied to hive). And there we write one event per file on s3 since s3 does not support reading a file in which data is still being written. It will write 2 log lines per query there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-842931967
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 4m 26s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 8m 44s | master passed |
| +1 :green_heart: | compile | 1m 38s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 28s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 41s | master passed |
| +1 :green_heart: | javadoc | 1m 36s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 17s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 44s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 3s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 55s | the patch passed |
| +1 :green_heart: | compile | 1m 3s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 3s | the patch passed |
| +1 :green_heart: | compile | 0m 53s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 53s | the patch passed |
| -0 :warning: | checkstyle | 0m 9s | tez-common: The patch generated 13 new + 0 unchanged - 0 fixed = 13 total (was 0) |
| -0 :warning: | checkstyle | 0m 8s | tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 49s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 46s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | findbugs | 2m 28s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 32s | tez-common in the patch passed. |
| -1 :x: | unit | 4m 16s | tez-dag in the patch failed. |
| +1 :green_heart: | unit | 0m 22s | tez-protobuf-history-plugin in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 39m 11s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | tez.dag.app.rm.TestTaskScheduler |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/120 |
| JIRA Issue | TEZ-4305 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux bd7a09ede325 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 01847814d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/8/artifact/out/diff-checkstyle-tez-common.txt |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/8/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt |
| unit | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/8/artifact/out/patch-unit-tez-dag.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/8/testReport/ |
| Max. process+thread count | 188 (vs. ulimit of 5500) |
| modules | C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/8/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-843287767
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 16m 52s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 1s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 4m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 8m 53s | master passed |
| +1 :green_heart: | compile | 1m 33s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 19s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 30s | master passed |
| +1 :green_heart: | javadoc | 1m 27s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 12s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 40s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 48s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 53s | the patch passed |
| +1 :green_heart: | compile | 0m 59s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 59s | the patch passed |
| +1 :green_heart: | compile | 0m 50s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 50s | the patch passed |
| -0 :warning: | checkstyle | 0m 8s | tez-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| -0 :warning: | checkstyle | 0m 7s | tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 50s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 46s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | findbugs | 2m 20s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 30s | tez-common in the patch passed. |
| +1 :green_heart: | unit | 4m 8s | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 0m 21s | tez-protobuf-history-plugin in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 53m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/120 |
| JIRA Issue | TEZ-4305 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 209bf44c28db 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 01847814d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/10/artifact/out/diff-checkstyle-tez-common.txt |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/10/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/10/testReport/ |
| Max. process+thread count | 215 (vs. ulimit of 5500) |
| modules | C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/10/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] kishendas commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
kishendas commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-826673196
This PR has dependency with https://github.com/apache/hadoop/pull/2949 . Will change the pom.xml to reflect the latest Hadoop version, whenever https://github.com/apache/hadoop/pull/2949 lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] kishendas commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
kishendas commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-843614921
@abstractdog Can you please approve and commit ? Finally tamed the checkstyle beast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
abstractdog commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-845179260
merged to master, thanks @kishendas for the patch and @harishjp for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] kishendas commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
kishendas commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-841623678
@abstractdog Please take a look. We can land this change without waiting for Hadoop 3.4.0 upgrade, as this change will work with both Hadoop 3.4.0 and previous versions. It's just that behavior will be different based on the Hadoop version we use with Tez.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] jteagles commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
jteagles commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-838957754
@kishendas, we will have to rethink this change to not break users who wish to run against other versions of Hadoop. @abstractdog, we may need to consider separate branch for Hadoop 3.3. There are several API changes and dependencies that are making supporting all of Hadoop 3.x difficult
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-841497951
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 12s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 5m 14s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 8m 30s | master passed |
| +1 :green_heart: | compile | 4m 6s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 3m 45s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 2m 34s | master passed |
| +1 :green_heart: | javadoc | 3m 45s | master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 9s | master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 6m 15s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +0 :ok: | findbugs | 6m 10s | root in master has 2 extant findbugs warnings. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 51s | the patch passed |
| +1 :green_heart: | compile | 3m 59s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 3m 59s | the patch passed |
| +1 :green_heart: | compile | 3m 41s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 3m 41s | the patch passed |
| -0 :warning: | checkstyle | 0m 25s | tez-common: The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0) |
| -0 :warning: | checkstyle | 0m 25s | tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| -0 :warning: | checkstyle | 0m 49s | root: The patch generated 11 new + 1 unchanged - 0 fixed = 12 total (was 1) |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 3m 54s | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 21s | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | findbugs | 9m 43s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 48s | tez-common in the patch passed. |
| +1 :green_heart: | unit | 4m 32s | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 0m 39s | tez-protobuf-history-plugin in the patch passed. |
| +1 :green_heart: | unit | 74m 25s | root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 46s | The patch does not generate ASF License warnings. |
| | | 157m 55s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/120 |
| JIRA Issue | TEZ-4305 |
| Optional Tests | dupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle |
| uname | Linux cc82bf444fb3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 01847814d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/artifact/out/diff-checkstyle-tez-common.txt |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/artifact/out/whitespace-eol.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/testReport/ |
| Max. process+thread count | 1409 (vs. ulimit of 5500) |
| modules | C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin . U: . |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
abstractdog commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-842181947
> @abstractdog Please take a look. We can land this change without waiting for Hadoop 3.4.0 upgrade, as this change will work with both Hadoop 3.4.0 and previous versions. It's just that behavior will be different based on the Hadoop version we use with Tez.
I agree, the latest version is even better because StreamCapabilites is already present in older Hadoop versions
once tez will depend on a hadoop version where HADOOP-17657 present (so writer implements StreamCapabilities), it will behave accordingly
nit: I was wondering if we can place a DEBUG message if hflush is skipped because it's not supported
this codepath is only called from finishCurrentDag, so once per dag, it won't spam the logs...
other than this nit, the patch looks good to me
cc: @harishjp if you have any comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] harishjp commented on pull request #120: TEZ-4305: Invoke hflush, only if the stream supports it
Posted by GitBox <gi...@apache.org>.
harishjp commented on pull request #120:
URL: https://github.com/apache/tez/pull/120#issuecomment-842251967
> nit: I was wondering if we can place a DEBUG message if hflush is skipped because it's not supported
> this codepath is only called from finishCurrentDag, so once per dag, it won't spam the logs...
> other than this nit, the patch looks good to me
> cc: @harishjp if you have any comments
This is fine in tez, we use the same code in hive (It has to be copied to hive). And there we write one event per file on s3 since s3 does not support reading a file in which data is still being written. It will write 2 log lines per query there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org