You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/11/03 02:32:57 UTC
[GitHub] [hadoop] viirya opened a new pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
viirya opened a new pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
This patch adds LZO Codec which includes `LzoCompressor` and `LzoDecompressor`.
See https://issues.apache.org/jira/browse/HADOOP-17124 for details.
### How was this patch tested?
Unit test.
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-960291528
Thanks @sunchao . So I think we verified the new Lzo compressor and GPL Lzo compressor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958607089
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958767912
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 14s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 12s | | trunk passed |
| +1 :green_heart: | compile | 23m 37s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 48s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 0s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 2s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 32s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 24m 23s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 26m 28s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 26m 28s | | the patch passed |
| +1 :green_heart: | compile | 21m 25s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 21m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 11s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/3/artifact/out/results-checkstyle-root.txt) | root: The patch generated 24 new + 69 unchanged - 0 fixed = 93 total (was 69) |
| +1 :green_heart: | mvnsite | 2m 9s | | the patch passed |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML file. |
| -1 :x: | javadoc | 1m 5s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javadoc | 1m 38s | [/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/3/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| +0 :ok: | spotbugs | 0m 31s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 25m 37s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 26s | | hadoop-project in the patch passed. |
| -1 :x: | unit | 17m 46s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not generate ASF License warnings. |
| | | 228m 17s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.io.file.tfile.TestTFileLzoCodecsStreams |
| | hadoop.io.file.tfile.TestTFileLzoCodecsByteArrays |
| | hadoop.io.compress.TestCodec |
| | hadoop.io.file.tfile.TestTFileSeqFileComparison |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3612 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle |
| uname | Linux 158ecb1201cb 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e7d98ab66585cbae11162ae605bf001839705be0 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/3/testReport/ |
| Max. process+thread count | 1261 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958606536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958606536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-960187534
@sunchao I built and installed this https://github.com/twitter/hadoop-lzo locally. You may need to revert https://github.com/apache/hadoop/pull/3612/commits/e7d98ab66585cbae11162ae605bf001839705be0 to use `0.4.21-SNAPSHOT` version built from the source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958607089
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 18s | | https://github.com/apache/hadoop/pull/3612 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/3612 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/1/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958653673
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 3s | | trunk passed |
| +1 :green_heart: | compile | 24m 12s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 7s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 4s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 2s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 32s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 24m 19s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 11s | [/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | compile | 0m 33s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 33s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 30s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 0m 30s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 30s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/artifact/out/results-checkstyle-root.txt) | root: The patch generated 24 new + 69 unchanged - 0 fixed = 93 total (was 69) |
| -1 :x: | mvnsite | 0m 12s | [/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML file. |
| -1 :x: | javadoc | 0m 12s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javadoc | 0m 12s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | hadoop-common in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +0 :ok: | spotbugs | 0m 13s | | hadoop-project has no data from spotbugs |
| -1 :x: | spotbugs | 0m 12s | [/patch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/artifact/out/patch-spotbugs-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | shadedclient | 1m 1s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 12s | | hadoop-project in the patch passed. |
| -1 :x: | unit | 0m 11s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not generate ASF License warnings. |
| | | 130m 7s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3612 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle |
| uname | Linux 9194c18eb7ec 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 426529446c52cdda37a9e8a3aab7bb9683dca263 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/testReport/ |
| Max. process+thread count | 518 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3612/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-959727355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958606819
`LzoCompressor` is ready for review. I will work on `LzoDecompressor` next.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya edited a comment on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-959725363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958606536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#discussion_r741581302
##########
File path: hadoop-common-project/hadoop-common/pom.xml
##########
@@ -375,6 +375,17 @@
<artifactId>lz4-java</artifactId>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>io.airlift</groupId>
+ <artifactId>aircompressor</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.hadoop.gplcompression</groupId>
+ <artifactId>hadoop-lzo</artifactId>
+ <version>0.4.21-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
Review comment:
Put it here for verifying the new Lzo codec only. We will remove it before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya edited a comment on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-959725363
Hmm, seems not work.
```
[ERROR] testLzoCodec(org.apache.hadoop.io.compress.TestCodec) Time elapsed: 0.004 s <<< ERROR!
java.lang.RuntimeException: native-lzo library not available
```
I checked `hadoop-lzo-0.4.20.jar`. It includes native library for `Linux-amd64-64` target, e.g.,
```
native/Linux-amd64-64/lib/libgplcompression.a
native/Linux-amd64-64/lib/libgplcompression.so
```
Maybe the CI is not the target?
So seems we cannot run the comparison test between the GPL lzo codec and this lzo codec on Hadoop CI.
I have run them locally to verify the comparison. The reviewers may build and install GPL lzo locally to run the test.
Once the reviewers think it is okay, I will remove GPL lzo codec stuffs.
@sunchao @dbtsai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958664918
Hmm, for the test, I built `hadoop-lzo` `0.4.21-SNAPSHOT` locally with native-lzo library for Mac OS. Switched to `0.4.20` for CI to see if it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya edited a comment on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-960187534
@sunchao It doesn't include native library for Mac OS X. So we need to built it.
I built and installed this https://github.com/twitter/hadoop-lzo locally. You may need to revert https://github.com/apache/hadoop/pull/3612/commits/e7d98ab66585cbae11162ae605bf001839705be0 to use `0.4.21-SNAPSHOT` version built from the source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-961244370
Yes looks OK so far. For reference, in the PR description could you add what other projects are using `aircompressor` for the same purpose (it'd be nice if you have a link to the related code)?
Feel free to update the PR and mark it as ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#discussion_r741581381
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/LzoCodec2.java
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.io.compress.lzo.LzoCompressor;
+import org.apache.hadoop.io.compress.lzo.LzoDecompressor;
+
+/**
+ * This class creates lzo compressors/decompressors.
+ */
+public class LzoCodec2 implements Configurable, CompressionCodec {
Review comment:
We will rename to `LzoCodec` before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#discussion_r741581518
##########
File path: hadoop-project/pom.xml
##########
@@ -2457,5 +2463,9 @@
</profiles>
<repositories>
+ <repository>
+ <id>twitter</id>
+ <url>https://maven.twttr.com/</url>
+ </repository>
Review comment:
We will remove this before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-960178116
@viirya can you share how to test this locally? I got the same error as above and curious why it didn't work even `hadoop-lzo` is already a test dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#discussion_r741581302
##########
File path: hadoop-common-project/hadoop-common/pom.xml
##########
@@ -375,6 +375,17 @@
<artifactId>lz4-java</artifactId>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>io.airlift</groupId>
+ <artifactId>aircompressor</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.hadoop.gplcompression</groupId>
+ <artifactId>hadoop-lzo</artifactId>
+ <version>0.4.21-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
Review comment:
Put it here for verifying the new Lzo codec only. We will remove it before merging.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/LzoCodec2.java
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.io.compress.lzo.LzoCompressor;
+import org.apache.hadoop.io.compress.lzo.LzoDecompressor;
+
+/**
+ * This class creates lzo compressors/decompressors.
+ */
+public class LzoCodec2 implements Configurable, CompressionCodec {
Review comment:
We will rename to `LzoCodec` before merging.
##########
File path: hadoop-project/pom.xml
##########
@@ -2457,5 +2463,9 @@
</profiles>
<repositories>
+ <repository>
+ <id>twitter</id>
+ <url>https://maven.twttr.com/</url>
+ </repository>
Review comment:
We will remove this before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya edited a comment on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-959725363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958606536
cc @sunchao @dbtsai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-959727355
Thanks @viirya , I'll take a look soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958607089
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-959727355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-960216919
Cool thanks. I verified locally and the relevant tests in `TestCodec` all passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-973344043
Thanks @sunchao !
I think the mostly famous usage of `aircompressor` is [trino](https://trino.io/). Trino uses the library for its Lz4Codec, LzoCodec, SnappyCodec, etc. The code link is:
https://github.com/trinodb/trino/blob/fe608f2723842037ff620d612a706900e79c52c8/lib/trino-rcfile/src/main/java/io/trino/rcfile/AircompressorCodecFactory.java
I also updated in the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958610739
Comparing with #2159, this adds `LzoCompressor` and `LzoDecompressor` (WIP) supports. I don't put this bridging classes with same name to `com.hadoop.compression.lzo.LzoCodec` etc. User might be unaware of the implicit change of codec, and I also think some users may still want to stick with original codec implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-959725363
Hmm, seems not work.
```
[ERROR] testLzoCodec(org.apache.hadoop.io.compress.TestCodec) Time elapsed: 0.004 s <<< ERROR!
java.lang.RuntimeException: native-lzo library not available
```
I checked `hadoop-lzo-0.4.20.jar`. It includes native library for `Linux-amd64-64` target, e.g.,
```
native/Linux-amd64-64/lib/libgplcompression.a
native/Linux-amd64-64/lib/libgplcompression.so
```
Maybe the CI is not the target?
So seems we cannot run the comparison test between the GPL lzo codec and this lzo codec.
I have run them locally to verify the comparison. The reviewers may build and install GPL lzo locally to run the test.
Once the reviewers think it is okay, I will remove GPL lzo codec stuffs.
@sunchao @dbtsai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-961244370
Yes looks OK so far. For reference, in the PR description could you add what other projects are using `aircompressor` for the same purpose (it'd be nice if you have a link to the related code)?
Feel free to update the PR and mark it as ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-958607089
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya edited a comment on pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#issuecomment-960187534
@sunchao I built and installed this https://github.com/twitter/hadoop-lzo locally. You may need to revert https://github.com/apache/hadoop/pull/3612/commits/e7d98ab66585cbae11162ae605bf001839705be0 to use `0.4.21-SNAPSHOT` version built from the source.
It doesn't include native library for Mac OS X.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #3612: WIP. HADOOP-17124. Support LZO Codec using aircompressor
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #3612:
URL: https://github.com/apache/hadoop/pull/3612#discussion_r741581302
##########
File path: hadoop-common-project/hadoop-common/pom.xml
##########
@@ -375,6 +375,17 @@
<artifactId>lz4-java</artifactId>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>io.airlift</groupId>
+ <artifactId>aircompressor</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.hadoop.gplcompression</groupId>
+ <artifactId>hadoop-lzo</artifactId>
+ <version>0.4.21-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
Review comment:
Put it here for verifying the new Lzo codec only. We will remove it before merging.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/LzoCodec2.java
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.io.compress.lzo.LzoCompressor;
+import org.apache.hadoop.io.compress.lzo.LzoDecompressor;
+
+/**
+ * This class creates lzo compressors/decompressors.
+ */
+public class LzoCodec2 implements Configurable, CompressionCodec {
Review comment:
We will rename to `LzoCodec` before merging.
##########
File path: hadoop-project/pom.xml
##########
@@ -2457,5 +2463,9 @@
</profiles>
<repositories>
+ <repository>
+ <id>twitter</id>
+ <url>https://maven.twttr.com/</url>
+ </repository>
Review comment:
We will remove this before merging.
##########
File path: hadoop-common-project/hadoop-common/pom.xml
##########
@@ -375,6 +375,17 @@
<artifactId>lz4-java</artifactId>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>io.airlift</groupId>
+ <artifactId>aircompressor</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.hadoop.gplcompression</groupId>
+ <artifactId>hadoop-lzo</artifactId>
+ <version>0.4.21-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
Review comment:
Put it here for verifying the new Lzo codec only. We will remove it before merging.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/LzoCodec2.java
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.io.compress.lzo.LzoCompressor;
+import org.apache.hadoop.io.compress.lzo.LzoDecompressor;
+
+/**
+ * This class creates lzo compressors/decompressors.
+ */
+public class LzoCodec2 implements Configurable, CompressionCodec {
Review comment:
We will rename to `LzoCodec` before merging.
##########
File path: hadoop-project/pom.xml
##########
@@ -2457,5 +2463,9 @@
</profiles>
<repositories>
+ <repository>
+ <id>twitter</id>
+ <url>https://maven.twttr.com/</url>
+ </repository>
Review comment:
We will remove this before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org