You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/12/08 14:55:30 UTC
[GitHub] [hadoop] virajjasani opened a new pull request #3768: HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
virajjasani opened a new pull request #3768:
URL: https://github.com/apache/hadoop/pull/3768
### Description of PR
As mentioned on HADOOP-17668, we can't upgrade hbase2 profile version beyond 2.2.4 until we either have hbase 2 artifacts available that are built with hadoop 3 profile by default or hbase 3 is rolled out (hbase 3 is compatible with hadoop 3 versions only).
This PR is to upgrade hbase2 profile version to 2.2.4 and also fix TestTimelineWriterHBaseDown to create connection only after mini cluster is up.
### How was this patch tested?
Local testing.
For unit testing, here is the test result with hbase2 profile:
```
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.996 s - in org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.288 s - in org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.485 s - in org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 201.434 s - in org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.86 s - in org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.672 s - in org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.734 s - in org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.285 s - in org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.799 s - in org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.664 s - in org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 103, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 09:33 min
[INFO] Finished at: 2021-12-08T19:32:42+05:30
[INFO] ------------------------------------------------------------------------
```
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #3768: HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #3768:
URL: https://github.com/apache/hadoop/pull/3768#discussion_r765137142
##########
File path: hadoop-project/pom.xml
##########
@@ -2428,9 +2428,10 @@
</activation>
<properties>
<hbase.version>${hbase.two.version}</hbase.version>
- <hbase-compatible-hadoop.version>3.0.0</hbase-compatible-hadoop.version>
+ <hbase-compatible-hadoop.version>2.8.5</hbase-compatible-hadoop.version>
Review comment:
Valid question. The earlier hbase version 2.0 is not only not so stable but also it just worked with Hadoop 3 version magically but as we upgrade HBase 2.x versions, they won't work by default with Hadoop 3 because as of today, hbase has only published hadoop 2 compatible hbase 2 artifacts by default. If we want to make hbase 2 work with hadoop 3, first we need to compile hbase 2 with hadoop-3.0 profile, only then we get hadoop 3 compatible hbase 2 jars. Hence, we should have anyways kept `hbase-compatible-hadoop.version` on Hadoop's 2.x release line but somehow we didn't and this worked with HBase 2.0.
I have provided detailed explanation of this also on HADOOP-17668. Comment [here](https://issues.apache.org/jira/browse/HADOOP-17668?focusedCommentId=17454172&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17454172) (hope I covered the right link, Jira interface seems to have been updated recently)
And yes @jojochuang's review would also be great, thanks for the reminder (my bad, I missed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #3768: HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #3768:
URL: https://github.com/apache/hadoop/pull/3768#discussion_r765500308
##########
File path: hadoop-project/pom.xml
##########
@@ -2428,9 +2428,10 @@
</activation>
<properties>
<hbase.version>${hbase.two.version}</hbase.version>
- <hbase-compatible-hadoop.version>3.0.0</hbase-compatible-hadoop.version>
+ <hbase-compatible-hadoop.version>2.8.5</hbase-compatible-hadoop.version>
Review comment:
hmm, I need to go through this and try, considering @aajisaka is convinced. I will back off for now.
You guys can proceed. Nothing blocking from my side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #3768: HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #3768:
URL: https://github.com/apache/hadoop/pull/3768#discussion_r767104461
##########
File path: hadoop-project/pom.xml
##########
@@ -2428,9 +2428,10 @@
</activation>
<properties>
<hbase.version>${hbase.two.version}</hbase.version>
- <hbase-compatible-hadoop.version>3.0.0</hbase-compatible-hadoop.version>
+ <hbase-compatible-hadoop.version>2.8.5</hbase-compatible-hadoop.version>
Review comment:
> Out of curiosity, why 2.8.5, nothing from 2.10 line or so?
To provide in-depth dependency view for this question: https://github.com/apache/hbase/blob/rel/2.2.4/pom.xml#L1373-L1378
```
<hadoop-two.version>2.8.5</hadoop-two.version>
<hadoop-three.version>3.1.2</hadoop-three.version>
<!-- These must be defined here for downstream build tools that don't look at profiles.
They ought to match the values found in our default hadoop profile, which is
currently "hadoop-2.0". See HBASE-15925 for more info. -->
<hadoop.version>${hadoop-two.version}</hadoop.version>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3768: HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3768:
URL: https://github.com/apache/hadoop/pull/3768#issuecomment-989055397
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 39s | | trunk passed |
| +1 :green_heart: | compile | 21m 50s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 9s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 43s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 20m 36s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 21m 14s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 21m 14s | | the patch passed |
| +1 :green_heart: | compile | 19m 9s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 35s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 25s | | the patch passed |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 38s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 20m 52s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 37s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 14m 29s | | hadoop-yarn-server-timelineservice-hbase-tests in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. |
| | | 198m 2s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3768/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3768 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle |
| uname | Linux 6b089725be82 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 38fbe6959c6453d0d08b58a58ea29c1d862cb8db |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3768/1/testReport/ |
| Max. process+thread count | 820 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3768/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #3768: HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
Posted by GitBox <gi...@apache.org>.
aajisaka merged pull request #3768:
URL: https://github.com/apache/hadoop/pull/3768
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #3768: HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #3768:
URL: https://github.com/apache/hadoop/pull/3768#discussion_r767104461
##########
File path: hadoop-project/pom.xml
##########
@@ -2428,9 +2428,10 @@
</activation>
<properties>
<hbase.version>${hbase.two.version}</hbase.version>
- <hbase-compatible-hadoop.version>3.0.0</hbase-compatible-hadoop.version>
+ <hbase-compatible-hadoop.version>2.8.5</hbase-compatible-hadoop.version>
Review comment:
> Out of curiosity, why 2.8.5, nothing from 2.10 line or so?
To provide in-depth dependency view for this question: https://github.com/apache/hbase/blob/rel/2.2.4/pom.xml#L1373-L1378
```
<hadoop-two.version>2.8.5</hadoop-two.version>
<hadoop-three.version>3.1.2</hadoop-three.version>
<!-- These must be defined here for downstream build tools that don't look at profiles.
They ought to match the values found in our default hadoop profile, which is
currently "hadoop-2.0". See HBASE-15925 for more info. -->
<hadoop.version>${hadoop-two.version}</hadoop.version>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3768: HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #3768:
URL: https://github.com/apache/hadoop/pull/3768#issuecomment-989057499
@aajisaka @ayushtkn Could you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #3768: HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #3768:
URL: https://github.com/apache/hadoop/pull/3768#issuecomment-992113039
Merged. Thank you @virajjasani and @ayushtkn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #3768: HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #3768:
URL: https://github.com/apache/hadoop/pull/3768#discussion_r765113619
##########
File path: hadoop-project/pom.xml
##########
@@ -2428,9 +2428,10 @@
</activation>
<properties>
<hbase.version>${hbase.two.version}</hbase.version>
- <hbase-compatible-hadoop.version>3.0.0</hbase-compatible-hadoop.version>
+ <hbase-compatible-hadoop.version>2.8.5</hbase-compatible-hadoop.version>
Review comment:
how is this upgrade downgrading the hadoop version? Earlier on a lower version we were good with 3.x and now to upgrade we are coming down.
Out of curiosity, why 2.8.5, nothing from 2.10 line or so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #3768: HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
Posted by GitBox <gi...@apache.org>.
jojochuang commented on a change in pull request #3768:
URL: https://github.com/apache/hadoop/pull/3768#discussion_r767537454
##########
File path: hadoop-project/pom.xml
##########
@@ -2428,9 +2428,10 @@
</activation>
<properties>
<hbase.version>${hbase.two.version}</hbase.version>
- <hbase-compatible-hadoop.version>3.0.0</hbase-compatible-hadoop.version>
+ <hbase-compatible-hadoop.version>2.8.5</hbase-compatible-hadoop.version>
Review comment:
Ah i see. Yes by default HBase 2 is built with Hadoop 2 profile. It is not until HBase 3.0 (being developed) that Hadoop 2 profile got removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org