You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2021/12/02 15:36:08 UTC
[GitHub] [hbase-connectors] LucaCanali opened a new pull request #88: [HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions.
LucaCanali opened a new pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88
This proposes to bump up the default versions used by HBase-connectors to recent versions, in particular for HBase, Spark, and Hadoop:
HBase is upgraded to 2.4.8, Spark to 3.1.2, Hadoop to 3.2.0.
This proposes also to make Spark 3 and Hadoop 3 default version (as opposed to Spark 2 and Hadoop 2).
Additional minor changes to the code are proposed/needed to compile the connector with HBase 2.4.8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] Apache-HBase commented on pull request #88: [HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-984754776
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 54s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | spotbugs | 0m 0s | spotbugs executables are not available. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 1m 13s | master passed |
| +1 :green_heart: | compile | 0m 47s | master passed |
| +1 :green_heart: | javadoc | 0m 48s | master passed |
| +1 :green_heart: | scaladoc | 0m 0s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 50s | the patch passed |
| +1 :green_heart: | compile | 0m 46s | the patch passed |
| +1 :green_heart: | javac | 0m 46s | the patch passed |
| +1 :green_heart: | scalac | 0m 46s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 46s | the patch passed |
| +1 :green_heart: | scaladoc | 0m 0s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 7m 12s | root in the patch passed. |
| | | 14m 58s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/1/artifact/yetus-precommit-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase-connectors/pull/88 |
| Optional Tests | dupname javac javadoc unit spotbugs findbugs compile xml markdownlint scalac scaladoc |
| uname | Linux b4309709c03d 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux |
| Build tool | hb_maven |
| Personality | dev-support/jenkins/hbase-personality.sh |
| git revision | master / 9a839b2 |
| Default Java | Oracle Corporation-1.8.0_282-b08 |
| whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/1/artifact/yetus-precommit-check/output/whitespace-eol.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/1/testReport/ |
| Max. process+thread count | 892 (vs. ulimit of 12500) |
| modules | C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/1/console |
| versions | git=2.20.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] Apache-HBase commented on pull request #88: [HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-1012015272
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 15s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | spotbugs | 0m 1s | spotbugs executables are not available. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 55s | master passed |
| +1 :green_heart: | compile | 0m 50s | master passed |
| +1 :green_heart: | javadoc | 0m 48s | master passed |
| +1 :green_heart: | scaladoc | 0m 0s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 53s | the patch passed |
| +1 :green_heart: | compile | 0m 48s | the patch passed |
| +1 :green_heart: | javac | 0m 48s | the patch passed |
| +1 :green_heart: | scalac | 0m 48s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed |
| +1 :green_heart: | scaladoc | 0m 0s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 7m 18s | root in the patch passed. |
| | | 17m 14s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/3/artifact/yetus-precommit-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase-connectors/pull/88 |
| Optional Tests | dupname javac javadoc unit spotbugs findbugs compile xml markdownlint scalac scaladoc |
| uname | Linux 1b6d27673a83 5.4.0-1047-aws #49~18.04.1-Ubuntu SMP Wed Apr 28 23:08:58 UTC 2021 x86_64 GNU/Linux |
| Build tool | hb_maven |
| Personality | dev-support/jenkins/hbase-personality.sh |
| git revision | master / 9a839b2 |
| Default Java | Oracle Corporation-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/3/testReport/ |
| Max. process+thread count | 917 (vs. ulimit of 12500) |
| modules | C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/3/console |
| versions | git=2.20.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] taklwu edited a comment on pull request #88: [HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
taklwu edited a comment on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-1011382325
@LucaCanali do you mind merging my changes from #89 to here? I will review yours instead, and that could help to move upgrade the forward .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] ianawilson commented on pull request #88: [HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
ianawilson commented on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-1012464281
This is great, thank you both @LucaCanali @taklwu! One question I've got: Is it possible for this update to be released and published to maven central? It seems like there are quite a few updates on this repository that haven't been released there yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] Apache-HBase commented on pull request #88: [HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-984895944
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 57s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | spotbugs | 0m 0s | spotbugs executables are not available. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 1m 28s | master passed |
| +1 :green_heart: | compile | 0m 51s | master passed |
| +1 :green_heart: | javadoc | 0m 50s | master passed |
| +1 :green_heart: | scaladoc | 0m 0s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 52s | the patch passed |
| +1 :green_heart: | compile | 0m 49s | the patch passed |
| +1 :green_heart: | javac | 0m 49s | the patch passed |
| +1 :green_heart: | scalac | 0m 49s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed |
| +1 :green_heart: | scaladoc | 0m 0s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 7m 24s | root in the patch passed. |
| | | 15m 41s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/2/artifact/yetus-precommit-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase-connectors/pull/88 |
| Optional Tests | dupname javac javadoc unit spotbugs findbugs compile xml markdownlint scalac scaladoc |
| uname | Linux f4ba024dd4d1 5.4.0-1047-aws #49~18.04.1-Ubuntu SMP Wed Apr 28 23:08:58 UTC 2021 x86_64 GNU/Linux |
| Build tool | hb_maven |
| Personality | dev-support/jenkins/hbase-personality.sh |
| git revision | master / 9a839b2 |
| Default Java | Oracle Corporation-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/2/testReport/ |
| Max. process+thread count | 916 (vs. ulimit of 12500) |
| modules | C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/2/console |
| versions | git=2.20.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] taklwu merged pull request #88: [HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
taklwu merged pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] ianawilson commented on pull request #88: [HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
ianawilson commented on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-1002309866
This is great. I ran into an issue with running this on AWS EMR and which provides HBase 2.4.1+ library code, and this had exactly the updates I needed.
I'd love to see this merged and packaged in a release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] taklwu commented on pull request #88: [HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
taklwu commented on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-1011382325
@ianawilson do you mind merging my changes from #89 to here? I will review yours instead, and that could help to move forward the upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] LucaCanali commented on pull request #88: [HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
LucaCanali commented on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-1009757866
Thanks @ianawilson for looking at this.
My thought and proposal with this is that if HBase 2.4.1 could afford a breaking change, that could reflect on the HBase connector too. Indeed this PR this requires HBase client 2.4.x to work. However, this is only for the HBase client version, it does not affect the server version, as many people are still using HBase versions older than 2.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] LucaCanali commented on pull request #88: [HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
LucaCanali commented on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-1012015703
Thanks @taklwu for comments and review.
I run my tests on Linux and they are all OK, however I have added your proposed changes to TestJavaHBaseContext.java too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] ianawilson commented on pull request #88: [HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
ianawilson commented on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-1002313179
One thing to think about with this:
HBase 2.4.1 introduced (via this PR https://github.com/apache/hbase/pull/2800) a breaking change for `bulkLoadThinRows()` by moving some of the static helpers from `HStore` to `StoreUtils` (this was precisely the problem that lead me to this PR).
This PR enables you to use this with HBase 2.4.1 and later, but will prevent you from using it with earlier HBase versions. I know you currently allow the HBase version to be configured via the system properties, but as-is this change won't work with the HBase library prior to 2.4.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] Apache-HBase commented on pull request #88: [HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-1012030168
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 44s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | spotbugs | 0m 0s | spotbugs executables are not available. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 47s | master passed |
| +1 :green_heart: | compile | 0m 48s | master passed |
| +1 :green_heart: | javadoc | 0m 48s | master passed |
| +1 :green_heart: | scaladoc | 0m 0s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 48s | the patch passed |
| +1 :green_heart: | compile | 0m 47s | the patch passed |
| +1 :green_heart: | javac | 0m 47s | the patch passed |
| +1 :green_heart: | scalac | 0m 47s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 45s | the patch passed |
| +1 :green_heart: | scaladoc | 0m 0s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 7m 37s | root in the patch passed. |
| | | 16m 59s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/4/artifact/yetus-precommit-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase-connectors/pull/88 |
| Optional Tests | dupname javac javadoc unit spotbugs findbugs compile xml markdownlint scalac scaladoc |
| uname | Linux 46e8c32c7378 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux |
| Build tool | hb_maven |
| Personality | dev-support/jenkins/hbase-personality.sh |
| git revision | master / 9a839b2 |
| Default Java | Oracle Corporation-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/4/testReport/ |
| Max. process+thread count | 903 (vs. ulimit of 12500) |
| modules | C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/4/console |
| versions | git=2.20.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] taklwu commented on a change in pull request #88: [HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#discussion_r783384657
##########
File path: pom.xml
##########
@@ -129,20 +129,20 @@
<compileSource>1.8</compileSource>
<java.min.version>${compileSource}</java.min.version>
<maven.min.version>3.5.0</maven.min.version>
- <hbase.version>2.2.2</hbase.version>
+ <hbase.version>2.4.8</hbase.version>
<exec.maven.version>1.6.0</exec.maven.version>
<audience-annotations.version>0.5.0</audience-annotations.version>
<junit.version>4.12</junit.version>
- <hbase-thirdparty.version>2.2.1</hbase-thirdparty.version>
+ <hbase-thirdparty.version>3.5.1</hbase-thirdparty.version>
Review comment:
do you see any problem of using 4.0.1? but in #89 , it does not seems to be a problem.
##########
File path: pom.xml
##########
@@ -129,20 +129,20 @@
<compileSource>1.8</compileSource>
<java.min.version>${compileSource}</java.min.version>
<maven.min.version>3.5.0</maven.min.version>
- <hbase.version>2.2.2</hbase.version>
+ <hbase.version>2.4.8</hbase.version>
<exec.maven.version>1.6.0</exec.maven.version>
<audience-annotations.version>0.5.0</audience-annotations.version>
<junit.version>4.12</junit.version>
- <hbase-thirdparty.version>2.2.1</hbase-thirdparty.version>
+ <hbase-thirdparty.version>3.5.1</hbase-thirdparty.version>
<hadoop-two.version>2.8.5</hadoop-two.version>
- <hadoop-three.version>3.0.3</hadoop-three.version>
- <hadoop.version>${hadoop-two.version}</hadoop.version>
+ <hadoop-three.version>3.2.0</hadoop-three.version>
+ <hadoop.version>${hadoop-three.version}</hadoop.version>
<slf4j.version>1.7.25</slf4j.version>
<log4j.version>1.2.17</log4j.version>
- <checkstyle.version>8.18</checkstyle.version>
- <maven.checkstyle.version>3.1.0</maven.checkstyle.version>
- <surefire.version>3.0.0-M4</surefire.version>
- <enforcer.version>3.0.0-M3</enforcer.version>
+ <checkstyle.version>8.45.1</checkstyle.version>
Review comment:
can you tell me why you chose this version instead of 8.28 that was used by the hbase master ?
##########
File path: spark/pom.xml
##########
@@ -44,13 +44,13 @@
<properties>
<protobuf.plugin.version>0.6.1</protobuf.plugin.version>
- <hbase-thirdparty.version>2.1.0</hbase-thirdparty.version>
- <jackson.version>2.9.10</jackson.version>
- <spark.version>2.4.0</spark.version>
+ <hbase-thirdparty.version>3.5.1</hbase-thirdparty.version>
Review comment:
remove this line
```suggestion
```
##########
File path: spark/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala
##########
@@ -902,8 +902,8 @@ class HBaseContext(@transient val sc: SparkContext,
tempConf.setFloat(HConstants.HFILE_BLOCK_CACHE_SIZE_KEY, 0.0f)
val contextBuilder = new HFileContextBuilder()
.withCompression(Algorithm.valueOf(familyOptions.compression))
- .withChecksumType(HStore.getChecksumType(conf))
- .withBytesPerCheckSum(HStore.getBytesPerChecksum(conf))
+ .withChecksumType(StoreUtils.getChecksumType(conf))
+ .withBytesPerCheckSum(StoreUtils.getBytesPerChecksum(conf))
Review comment:
```suggestion
// HBASE-25249 introduced an incompatible change in the IA.Private HStore and StoreUtils
// so here, we directly use conf.get for CheckSumType and BytesPerCheckSum to make it
// compatible between hbase 2.3.x and 2.4.x
val contextBuilder = new HFileContextBuilder()
.withCompression(Algorithm.valueOf(familyOptions.compression))
// ChecksumType.nameToType is still an IA.Private Utils, but it's unlikely to be changed.
.withChecksumType(ChecksumType
.nameToType(conf.get(HConstants.CHECKSUM_TYPE_NAME,
ChecksumType.getDefaultChecksumType.getName)))
.withCellComparator(CellComparator.getInstance())
.withBytesPerCheckSum(conf.getInt(HConstants.BYTES_PER_CHECKSUM,
HFile.DEFAULT_BYTES_PER_CHECKSUM))
.withBlockSize(familyOptions.blockSize)
```
basically I want to make it compatible with hbase-2.3.7 as well
##########
File path: spark/README.md
##########
@@ -18,19 +18,25 @@ limitations under the License.
# Apache HBase™ Spark Connector
-## Scala and Spark Versions
+## Spark, Scala and Configurable Options
-To generate an artifact for a different [spark version](https://mvnrepository.com/artifact/org.apache.spark/spark-core) and/or [scala version](https://www.scala-lang.org/download/all.html), pass command-line options as follows (changing version numbers appropriately):
+To generate an artifact for a different [Spark version](https://mvnrepository.com/artifact/org.apache.spark/spark-core) and/or [Scala version](https://www.scala-lang.org/download/all.html),
+[Hadoop version](https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-core), or [HBase version](https://mvnrepository.com/artifact/org.apache.hbase/hbase), pass command-line options as follows (changing version numbers appropriately):
```
-$ mvn -Dspark.version=2.2.2 -Dscala.version=2.11.7 -Dscala.binary.version=2.11 clean install
+$ mvn -Dspark.version=3.1.2 -Dscala.version=2.12.10 -Dhadoop-three.version=3.2.0 -Dscala.binary.version=2.12 -Dhbase.version=2.4.8 clean install
```
----
-To build the connector with Spark 3.0, compile it with scala 2.12.
-Additional configurations that you can customize are the Spark version, HBase version, and Hadoop version.
-Example:
+Note: to build the connector with Spark 2.x, compile it with `-Dscala.binary.version=2.11` and use the profile `-Dhadoop.profile=2.0`
+
+## Configuration and Installation
+**Client-side** (Spark) configuration:
+- The HBase configuration file `hbase-site.xml` should be made available to Spark, it can be copied to `$SPARK_CONF_DIR` (default is $SPARK_HOME/conf`)
+
+**Server-side** (HBase region servers) configuration:
+- The following jars needs to be in the CLASSPATH of the HBase region servers:
Review comment:
```suggestion
- The following jars need to be in the CLASSPATH of the HBase region servers:
```
##########
File path: pom.xml
##########
@@ -129,20 +129,20 @@
<compileSource>1.8</compileSource>
<java.min.version>${compileSource}</java.min.version>
<maven.min.version>3.5.0</maven.min.version>
- <hbase.version>2.2.2</hbase.version>
+ <hbase.version>2.4.8</hbase.version>
Review comment:
use 2.4.9
##########
File path: spark/hbase-spark/pom.xml
##########
@@ -345,20 +347,16 @@
</dependency>
</dependencies>
</profile>
- <!--
- profile for building against Hadoop 3.0.x. Activate using:
- mvn -Dhadoop.profile=3.0
- -->
+ <!-- profile against Hadoop 3.x: This is the default. -->
<profile>
<id>hadoop-3.0</id>
<activation>
<property>
- <name>hadoop.profile</name>
- <value>3.0</value>
+ <name>!hadoop.profile</name>
</property>
</activation>
<properties>
- <hadoop.version>3.0</hadoop.version>
+ <hadoop.version>3.2.0</hadoop.version>
Review comment:
```suggestion
<hadoop.version>${hadoop-three.version}</hadoop.version>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] taklwu edited a comment on pull request #88: [HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
taklwu edited a comment on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-1011382325
@LucaCanali do you mind merging my changes from #89 to here? I will review yours instead, and that could help to move forward the upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase-connectors] ianawilson commented on pull request #88: [HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions.
Posted by GitBox <gi...@apache.org>.
ianawilson commented on pull request #88:
URL: https://github.com/apache/hbase-connectors/pull/88#issuecomment-1011288094
@LucaCanali FYI, looks like there's a PR with a similar intent and slightly different approach https://github.com/apache/hbase-connectors/pull/89
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org