You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by GitBox <gi...@apache.org> on 2021/04/08 07:09:01 UTC
[GitHub] [tez] jshmchenxi opened a new pull request #117: TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null
jshmchenxi opened a new pull request #117:
URL: https://github.com/apache/tez/pull/117
Issue: [TEZ-4302](https://issues.apache.org/jira/browse/TEZ-4302) NullPointerException in CodecUtils when bufferSizeProp is null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog commented on a change in pull request #117: TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null
Posted by GitBox <gi...@apache.org>.
abstractdog commented on a change in pull request #117:
URL: https://github.com/apache/tez/pull/117#discussion_r609483156
##########
File path: tez-runtime-library/src/main/java/org/apache/tez/runtime/library/utils/CodecUtils.java
##########
@@ -78,7 +78,8 @@ public static InputStream getDecompressedInputStreamWithBufferSize(CompressionCo
throws IOException {
String bufferSizeProp = TezRuntimeUtils.getBufferSizeProperty(codec);
Configurable configurableCodec = (Configurable) codec;
- int originalSize = configurableCodec.getConf().getInt(bufferSizeProp, DEFAULT_BUFFER_SIZE);
+ int originalSize = bufferSizeProp == null ? DEFAULT_BUFFER_SIZE :
Review comment:
thanks @jshmchenxi, this looks like an important bugfix
before merging this, could you please clarify what kind of coded have you used?
if bufferSizeProp is null, it means that the codec is not supported by [TezRuntimeUtils.getBufferSizeProperty](https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/TezRuntimeUtils.java#L269)
if it's a codec supported by hadoop, we should add it, otherwise the DEFAULT_BUFFER_SIZE fallback makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #117: TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #117:
URL: https://github.com/apache/tez/pull/117#issuecomment-815785697
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 12m 58s | master passed |
| +1 :green_heart: | compile | 0m 34s | master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 33s | master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 5s | master passed |
| +1 :green_heart: | javadoc | 0m 40s | master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +0 :ok: | spotbugs | 1m 25s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 23s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 21s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| +1 :green_heart: | compile | 0m 19s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 19s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 14s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 18s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 16s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | findbugs | 0m 52s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 5m 9s | tez-runtime-library in the patch passed. |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 27m 10s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-117/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/117 |
| JIRA Issue | TEZ-4302 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 80b0de175ea2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 2b671d621 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-117/2/testReport/ |
| Max. process+thread count | 121 (vs. ulimit of 5500) |
| modules | C: tez-runtime-library U: tez-runtime-library |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-117/2/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog merged pull request #117: TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null
Posted by GitBox <gi...@apache.org>.
abstractdog merged pull request #117:
URL: https://github.com/apache/tez/pull/117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog commented on a change in pull request #117: TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null
Posted by GitBox <gi...@apache.org>.
abstractdog commented on a change in pull request #117:
URL: https://github.com/apache/tez/pull/117#discussion_r609483156
##########
File path: tez-runtime-library/src/main/java/org/apache/tez/runtime/library/utils/CodecUtils.java
##########
@@ -78,7 +78,8 @@ public static InputStream getDecompressedInputStreamWithBufferSize(CompressionCo
throws IOException {
String bufferSizeProp = TezRuntimeUtils.getBufferSizeProperty(codec);
Configurable configurableCodec = (Configurable) codec;
- int originalSize = configurableCodec.getConf().getInt(bufferSizeProp, DEFAULT_BUFFER_SIZE);
+ int originalSize = bufferSizeProp == null ? DEFAULT_BUFFER_SIZE :
Review comment:
thanks @jshmchenxi, this looks like an important bugfix
before merging this, could you please clarify what kind of codec have you used?
if bufferSizeProp is null, it means that the codec is not supported by [TezRuntimeUtils.getBufferSizeProperty](https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/TezRuntimeUtils.java#L269)
if it's a codec supported by hadoop, we should add it, otherwise the DEFAULT_BUFFER_SIZE fallback makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] jshmchenxi commented on a change in pull request #117: TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null
Posted by GitBox <gi...@apache.org>.
jshmchenxi commented on a change in pull request #117:
URL: https://github.com/apache/tez/pull/117#discussion_r609635657
##########
File path: tez-runtime-library/src/main/java/org/apache/tez/runtime/library/utils/CodecUtils.java
##########
@@ -78,7 +78,8 @@ public static InputStream getDecompressedInputStreamWithBufferSize(CompressionCo
throws IOException {
String bufferSizeProp = TezRuntimeUtils.getBufferSizeProperty(codec);
Configurable configurableCodec = (Configurable) codec;
- int originalSize = configurableCodec.getConf().getInt(bufferSizeProp, DEFAULT_BUFFER_SIZE);
+ int originalSize = bufferSizeProp == null ? DEFAULT_BUFFER_SIZE :
Review comment:
@abstractdog thanks for the quick reply! I've updated the patch. In our environment the codec is GzipCodec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] hadoop-yetus commented on pull request #117: TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #117:
URL: https://github.com/apache/tez/pull/117#issuecomment-815536912
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 12m 56s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 12m 54s | master passed |
| +1 :green_heart: | compile | 0m 35s | master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 33s | master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 8s | master passed |
| +1 :green_heart: | javadoc | 0m 40s | master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +0 :ok: | spotbugs | 1m 25s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 23s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 21s | the patch passed |
| +1 :green_heart: | compile | 0m 22s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 22s | the patch passed |
| +1 :green_heart: | compile | 0m 19s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 19s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 15s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 19s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 16s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | findbugs | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 5m 18s | tez-runtime-library in the patch passed. |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 39m 55s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-117/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/117 |
| JIRA Issue | TEZ-4302 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux ff7d77fb1ce3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 2b671d621 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-117/1/testReport/ |
| Max. process+thread count | 116 (vs. ulimit of 5500) |
| modules | C: tez-runtime-library U: tez-runtime-library |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-117/1/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] jteagles commented on a change in pull request #117: TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null
Posted by GitBox <gi...@apache.org>.
jteagles commented on a change in pull request #117:
URL: https://github.com/apache/tez/pull/117#discussion_r610665429
##########
File path: tez-runtime-library/src/main/java/org/apache/tez/runtime/library/utils/CodecUtils.java
##########
@@ -78,7 +78,8 @@ public static InputStream getDecompressedInputStreamWithBufferSize(CompressionCo
throws IOException {
String bufferSizeProp = TezRuntimeUtils.getBufferSizeProperty(codec);
Configurable configurableCodec = (Configurable) codec;
- int originalSize = configurableCodec.getConf().getInt(bufferSizeProp, DEFAULT_BUFFER_SIZE);
+ int originalSize = bufferSizeProp == null ? DEFAULT_BUFFER_SIZE :
Review comment:
@abstractdog, I have added contributor rights to @jshmchenxi to the Tez project in jira. I have also added you (@abstractdog) to the administrator role to the Tez which grants permissions to add contributors and other project level changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog commented on a change in pull request #117: TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null
Posted by GitBox <gi...@apache.org>.
abstractdog commented on a change in pull request #117:
URL: https://github.com/apache/tez/pull/117#discussion_r609906867
##########
File path: tez-runtime-library/src/main/java/org/apache/tez/runtime/library/utils/CodecUtils.java
##########
@@ -78,7 +78,8 @@ public static InputStream getDecompressedInputStreamWithBufferSize(CompressionCo
throws IOException {
String bufferSizeProp = TezRuntimeUtils.getBufferSizeProperty(codec);
Configurable configurableCodec = (Configurable) codec;
- int originalSize = configurableCodec.getConf().getInt(bufferSizeProp, DEFAULT_BUFFER_SIZE);
+ int originalSize = bufferSizeProp == null ? DEFAULT_BUFFER_SIZE :
Review comment:
thanks for the fix, I'm about to merge this soon if no objections
@jteagles: could you please contributor rights to @jshmchenxi to the Tez project in Jira?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org