You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/10/28 00:04:43 UTC
[GitHub] [hbase] sandeepvinayak opened a new pull request #2591: HBASE-24859: Improve the storage cost for HBase map reduce table splits
sandeepvinayak opened a new pull request #2591:
URL: https://github.com/apache/hbase/pull/2591
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#issuecomment-717775168
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 42s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ branch-1 Compile Tests _ |
| +1 :green_heart: | mvninstall | 9m 53s | branch-1 passed |
| +1 :green_heart: | compile | 0m 44s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | compile | 0m 46s | branch-1 passed with JDK v1.7.0_272 |
| +1 :green_heart: | checkstyle | 1m 44s | branch-1 passed |
| +1 :green_heart: | shadedjars | 3m 4s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 48s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 42s | branch-1 passed with JDK v1.7.0_272 |
| +0 :ok: | spotbugs | 3m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 4s | branch-1 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 57s | the patch passed |
| +1 :green_heart: | compile | 0m 43s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javac | 0m 43s | the patch passed |
| +1 :green_heart: | compile | 0m 43s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | javac | 0m 43s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 34s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 2m 50s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 4m 37s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. |
| +1 :green_heart: | javadoc | 0m 34s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 42s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | findbugs | 2m 56s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 106m 51s | hbase-server in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 149m 13s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2591 |
| JIRA Issue | HBASE-24859 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 060e7bb3d2a2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2591/out/precommit/personality/provided.sh |
| git revision | branch-1 / b30d1d1 |
| Default Java | 1.7.0_272 |
| Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/3/testReport/ |
| Max. process+thread count | 3930 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/3/console |
| versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
sandeepvinayak commented on a change in pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#discussion_r513795763
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java
##########
@@ -184,9 +184,24 @@ public TableSplit(final byte [] tableName, byte[] startRow, byte[] endRow,
* @param startRow The start row of the split.
* @param endRow The end row of the split.
* @param location The location of the region.
+ * @param encodedRegionName The region ID.
+ * @param length Size of region in bytes
*/
public TableSplit(TableName tableName, byte[] startRow, byte[] endRow,
- final String location) {
+ final String location, final String encodedRegionName, long length) {
+ this(tableName, null, startRow, endRow, location, encodedRegionName, length);
Review comment:
@bharathv I think the separate constructor is more intuitive that it is okay to not have scan object? What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
saintstack commented on pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#issuecomment-718094256
@bharathv thank you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#discussion_r513179886
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java
##########
@@ -323,7 +323,7 @@ public boolean nextKeyValue() throws IOException, InterruptedException {
}
List<InputSplit> splits = new ArrayList<>(1);
long regionSize = sizeCalculator.getRegionSize(regLoc.getRegionInfo().getRegionName());
- TableSplit split = new TableSplit(tableName, scan,
+ TableSplit split = new TableSplit(tableName,
Review comment:
So, we do not pass the scan object and then we save a bunch of memory? The TableSplit doesn't carry around the heavy scan instance anymore? Is that it? Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
sandeepvinayak commented on a change in pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#discussion_r513830335
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
##########
@@ -271,6 +272,11 @@ public void testNumOfSplits(int splitsPerRegion, int expectedNumOfSplits) throws
TableInputFormat tif = new TableInputFormat();
tif.setConf(job.getConfiguration());
List<InputSplit> splits = tif.getSplits(job);
+ for (InputSplit split : splits) {
+ TableSplit tableSplit = (TableSplit) split;
+ Assert.assertEquals(tableSplit.getScan().getStartRow(), HConstants.EMPTY_START_ROW);
Review comment:
@bharathv The reason I am asserting on this is when we serialize null scan object, we set the default object. That's why we don't get the empty string back but with the defaut values. I am asserting here on default values.
https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java#L1126
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Reidddddd commented on a change in pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
Reidddddd commented on a change in pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#discussion_r514092739
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java
##########
@@ -184,9 +184,24 @@ public TableSplit(final byte [] tableName, byte[] startRow, byte[] endRow,
* @param startRow The start row of the split.
* @param endRow The end row of the split.
* @param location The location of the region.
+ * @param encodedRegionName The region ID.
+ * @param length Size of region in bytes
*/
public TableSplit(TableName tableName, byte[] startRow, byte[] endRow,
- final String location) {
+ final String location, final String encodedRegionName, long length) {
+ this(tableName, null, startRow, endRow, location, encodedRegionName, length);
Review comment:
can't we ignore the scan assignment, just using `this.scan = ""` anyway, instead of the following:
```
try {
this.scan =
(null == scan) ? "" : TableMapReduceUtil.convertScanToString(scan);
} catch (IOException e) {
LOG.warn("Failed to convert Scan to String", e);
}
```
I also prefer not to add a new constructor.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java
##########
@@ -184,9 +184,24 @@ public TableSplit(final byte [] tableName, byte[] startRow, byte[] endRow,
* @param startRow The start row of the split.
* @param endRow The end row of the split.
* @param location The location of the region.
+ * @param encodedRegionName The region ID.
+ * @param length Size of region in bytes
*/
public TableSplit(TableName tableName, byte[] startRow, byte[] endRow,
- final String location) {
+ final String location, final String encodedRegionName, long length) {
+ this(tableName, null, startRow, endRow, location, encodedRegionName, length);
Review comment:
iff we make sure the scan is useless here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2591: HBASE-24859: Optimize in-memory representation for HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#issuecomment-717657954
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 6m 52s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ branch-1 Compile Tests _ |
| +1 :green_heart: | mvninstall | 9m 50s | branch-1 passed |
| +1 :green_heart: | compile | 0m 42s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | compile | 0m 46s | branch-1 passed with JDK v1.7.0_272 |
| +1 :green_heart: | checkstyle | 1m 41s | branch-1 passed |
| +1 :green_heart: | shadedjars | 3m 5s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 49s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 41s | branch-1 passed with JDK v1.7.0_272 |
| +0 :ok: | spotbugs | 3m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 2s | branch-1 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 58s | the patch passed |
| +1 :green_heart: | compile | 0m 42s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javac | 0m 42s | the patch passed |
| +1 :green_heart: | compile | 0m 45s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | javac | 0m 45s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 29s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 2m 51s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 4m 41s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. |
| -1 :x: | javadoc | 0m 32s | hbase-server-jdk1.8.0_262 with JDK v1.8.0_262 generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8) |
| -1 :x: | javadoc | 0m 41s | hbase-server-jdk1.7.0_272 with JDK v1.7.0_272 generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8) |
| +1 :green_heart: | findbugs | 2m 55s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 105m 17s | hbase-server in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 153m 44s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2591 |
| JIRA Issue | HBASE-24859 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 8bd2f8cee8c6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2591/out/precommit/personality/provided.sh |
| git revision | branch-1 / b30d1d1 |
| Default Java | 1.7.0_272 |
| Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/1/artifact/out/diff-javadoc-javadoc-hbase-server-jdk1.8.0_262.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/1/artifact/out/diff-javadoc-javadoc-hbase-server-jdk1.7.0_272.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/1/testReport/ |
| Max. process+thread count | 3942 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/1/console |
| versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
bharathv commented on pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#issuecomment-718092878
@saintstack I can sign-off on this change, I have some context on the bug, will take a look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#issuecomment-718324114
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 22s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ branch-1 Compile Tests _ |
| +1 :green_heart: | mvninstall | 9m 48s | branch-1 passed |
| +1 :green_heart: | compile | 0m 43s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | compile | 0m 45s | branch-1 passed with JDK v1.7.0_272 |
| +1 :green_heart: | checkstyle | 1m 42s | branch-1 passed |
| +1 :green_heart: | shadedjars | 3m 1s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 47s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 40s | branch-1 passed with JDK v1.7.0_272 |
| +0 :ok: | spotbugs | 3m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 3s | branch-1 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 56s | the patch passed |
| +1 :green_heart: | compile | 0m 41s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javac | 0m 41s | the patch passed |
| +1 :green_heart: | compile | 0m 45s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | javac | 0m 45s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 34s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 2m 52s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 4m 45s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. |
| +1 :green_heart: | javadoc | 0m 31s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 40s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | findbugs | 3m 0s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 125m 5s | hbase-server in the patch failed. |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 167m 47s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2591 |
| JIRA Issue | HBASE-24859 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux bb5cb308c89b 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2591/out/precommit/personality/provided.sh |
| git revision | branch-1 / b30d1d1 |
| Default Java | 1.7.0_272 |
| Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/4/artifact/out/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/4/testReport/ |
| Max. process+thread count | 4579 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/4/console |
| versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
sandeepvinayak commented on a change in pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#discussion_r513599178
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java
##########
@@ -323,7 +323,7 @@ public boolean nextKeyValue() throws IOException, InterruptedException {
}
List<InputSplit> splits = new ArrayList<>(1);
long regionSize = sizeCalculator.getRegionSize(regLoc.getRegionInfo().getRegionName());
- TableSplit split = new TableSplit(tableName, scan,
+ TableSplit split = new TableSplit(tableName,
Review comment:
@saintstack that is correct! If you see the jira for description, there is a heap dump screenshots which shows the scan object may occupy much memory in case of tables with large number of regions. This patch just fix the TableInputFormat for single table where we don’t use the scan object from TableSplit since we use it from MR Job conf directly. There should be another patch to fix the similar fix with more code changes for MultiTableInputFormat. I will try to fix that in a separate patch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#issuecomment-719067255
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 49s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ branch-1 Compile Tests _ |
| +1 :green_heart: | mvninstall | 10m 7s | branch-1 passed |
| +1 :green_heart: | compile | 0m 39s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | compile | 0m 45s | branch-1 passed with JDK v1.7.0_272 |
| +1 :green_heart: | checkstyle | 1m 44s | branch-1 passed |
| +1 :green_heart: | shadedjars | 3m 4s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 49s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 42s | branch-1 passed with JDK v1.7.0_272 |
| +0 :ok: | spotbugs | 3m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 0s | branch-1 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 53s | the patch passed |
| +1 :green_heart: | compile | 0m 40s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javac | 0m 40s | the patch passed |
| +1 :green_heart: | compile | 0m 44s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | javac | 0m 44s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 2m 51s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 4m 35s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. |
| +1 :green_heart: | javadoc | 0m 31s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 41s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | findbugs | 2m 50s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 123m 31s | hbase-server in the patch failed. |
| +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. |
| | | 167m 35s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2591 |
| JIRA Issue | HBASE-24859 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux fdf2c3c397d5 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2591/out/precommit/personality/provided.sh |
| git revision | branch-1 / b30d1d1 |
| Default Java | 1.7.0_272 |
| Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/6/artifact/out/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/6/testReport/ |
| Max. process+thread count | 4046 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/6/console |
| versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#discussion_r514426736
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
##########
@@ -271,6 +272,11 @@ public void testNumOfSplits(int splitsPerRegion, int expectedNumOfSplits) throws
TableInputFormat tif = new TableInputFormat();
tif.setConf(job.getConfiguration());
List<InputSplit> splits = tif.getSplits(job);
+ for (InputSplit split : splits) {
+ TableSplit tableSplit = (TableSplit) split;
+ Assert.assertEquals(tableSplit.getScan().getStartRow(), HConstants.EMPTY_START_ROW);
Review comment:
Ya my point is, that is a weak check. You wouldn't know if that scan is an empty scan or a default scan. Change the code to something like this and you can see there is a bug ...
```
for (InputSplit split : splits) {
TableSplit tableSplit = (TableSplit) split;
assertTrue(tableSplit.getScanAsString().isEmpty());
}
In table split code, add this...
public String getScanAsString() {
return scan;
}
```
Run this test
```
@Test
public void testGetSplits() throws IOException, InterruptedException, ClassNotFoundException {
testNumOfSplits(1, 26);
testNumOfSplits(3, 78); <====
}
```
If number of splits per region is > 1, your patch didn't handle the TableSplit constructor in this call createNInputSplitsUniform(). Your test still passes because you are asserting on a default Scan object and not the fact that it shouldn't be set in the first place.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java
##########
@@ -184,9 +184,24 @@ public TableSplit(final byte [] tableName, byte[] startRow, byte[] endRow,
* @param startRow The start row of the split.
* @param endRow The end row of the split.
* @param location The location of the region.
+ * @param encodedRegionName The region ID.
+ * @param length Size of region in bytes
*/
public TableSplit(TableName tableName, byte[] startRow, byte[] endRow,
- final String location) {
+ final String location, final String encodedRegionName, long length) {
+ this(tableName, null, startRow, endRow, location, encodedRegionName, length);
Review comment:
> I think the separate constructor is more intuitive that it is okay to not have scan object? What do you think?
IMHO more constructors is less readability. In this case we have 8 constructors for TableSplit. If I were to use it, I'd confused unless I see the usages of each of these. Either we should have a [fluent interface](https://en.wikipedia.org/wiki/Fluent_interface) or fewer constructors is what I think. Subjective of course.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
sandeepvinayak commented on a change in pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#discussion_r514528418
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java
##########
@@ -184,9 +184,24 @@ public TableSplit(final byte [] tableName, byte[] startRow, byte[] endRow,
* @param startRow The start row of the split.
* @param endRow The end row of the split.
* @param location The location of the region.
+ * @param encodedRegionName The region ID.
+ * @param length Size of region in bytes
*/
public TableSplit(TableName tableName, byte[] startRow, byte[] endRow,
- final String location) {
+ final String location, final String encodedRegionName, long length) {
+ this(tableName, null, startRow, endRow, location, encodedRegionName, length);
Review comment:
@Reidddddd Scan is not useless for all the input formats of map-reduce, it is for the `TableInputFormat`.
We do need the scan object for `MultiTableInputFormat`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
sandeepvinayak commented on pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#issuecomment-718102719
Created a separate jira https://issues.apache.org/jira/browse/HBASE-25226 for optimizing the memory consumption in `MultiTableInputFormat`.
@bharathv @saintstack
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#issuecomment-717714818
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 12m 24s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ branch-1 Compile Tests _ |
| +1 :green_heart: | mvninstall | 9m 37s | branch-1 passed |
| +1 :green_heart: | compile | 0m 41s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | compile | 0m 44s | branch-1 passed with JDK v1.7.0_272 |
| +1 :green_heart: | checkstyle | 1m 40s | branch-1 passed |
| +1 :green_heart: | shadedjars | 3m 0s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 49s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 41s | branch-1 passed with JDK v1.7.0_272 |
| +0 :ok: | spotbugs | 3m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 1s | branch-1 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 55s | the patch passed |
| +1 :green_heart: | compile | 0m 40s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javac | 0m 40s | the patch passed |
| +1 :green_heart: | compile | 0m 46s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | javac | 0m 46s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 36s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 2m 56s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 4m 44s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. |
| -1 :x: | javadoc | 0m 33s | hbase-server-jdk1.8.0_262 with JDK v1.8.0_262 generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8) |
| -1 :x: | javadoc | 0m 42s | hbase-server-jdk1.7.0_272 with JDK v1.7.0_272 generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8) |
| +1 :green_heart: | findbugs | 2m 58s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 137m 38s | hbase-server in the patch failed. |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 191m 20s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2591 |
| JIRA Issue | HBASE-24859 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 158ee93df235 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2591/out/precommit/personality/provided.sh |
| git revision | branch-1 / b30d1d1 |
| Default Java | 1.7.0_272 |
| Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/2/artifact/out/diff-javadoc-javadoc-hbase-server-jdk1.8.0_262.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/2/artifact/out/diff-javadoc-javadoc-hbase-server-jdk1.7.0_272.txt |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/2/artifact/out/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/2/testReport/ |
| Max. process+thread count | 4098 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/2/console |
| versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
sandeepvinayak commented on a change in pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#discussion_r514525613
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
##########
@@ -271,6 +272,11 @@ public void testNumOfSplits(int splitsPerRegion, int expectedNumOfSplits) throws
TableInputFormat tif = new TableInputFormat();
tif.setConf(job.getConfiguration());
List<InputSplit> splits = tif.getSplits(job);
+ for (InputSplit split : splits) {
+ TableSplit tableSplit = (TableSplit) split;
+ Assert.assertEquals(tableSplit.getScan().getStartRow(), HConstants.EMPTY_START_ROW);
Review comment:
I see your point, thanks !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#discussion_r513757980
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java
##########
@@ -184,9 +184,24 @@ public TableSplit(final byte [] tableName, byte[] startRow, byte[] endRow,
* @param startRow The start row of the split.
* @param endRow The end row of the split.
* @param location The location of the region.
+ * @param encodedRegionName The region ID.
+ * @param length Size of region in bytes
*/
public TableSplit(TableName tableName, byte[] startRow, byte[] endRow,
- final String location) {
+ final String location, final String encodedRegionName, long length) {
+ this(tableName, null, startRow, endRow, location, encodedRegionName, length);
Review comment:
Why do we need a new constructor? Just pass null wherever it is not needed?
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
##########
@@ -271,6 +272,11 @@ public void testNumOfSplits(int splitsPerRegion, int expectedNumOfSplits) throws
TableInputFormat tif = new TableInputFormat();
tif.setConf(job.getConfiguration());
List<InputSplit> splits = tif.getSplits(job);
+ for (InputSplit split : splits) {
+ TableSplit tableSplit = (TableSplit) split;
+ Assert.assertEquals(tableSplit.getScan().getStartRow(), HConstants.EMPTY_START_ROW);
Review comment:
nit: I think these checks are not tight enough since they don't differentiate between a full table scan object and a no scan object, instead I think it should be something like
assertTrue(tableSplit.getScanAsString().isEmpty()) or some such...
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java
##########
@@ -184,9 +184,24 @@ public TableSplit(final byte [] tableName, byte[] startRow, byte[] endRow,
* @param startRow The start row of the split.
* @param endRow The end row of the split.
* @param location The location of the region.
+ * @param encodedRegionName The region ID.
+ * @param length Size of region in bytes
*/
public TableSplit(TableName tableName, byte[] startRow, byte[] endRow,
- final String location) {
+ final String location, final String encodedRegionName, long length) {
+ this(tableName, null, startRow, endRow, location, encodedRegionName, length);
+ }
+
+ /**
+ * Creates a new instance without a scanner.
Review comment:
Mind adding a javadoc for "scan" object? What it means to be null..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#issuecomment-719121119
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 13s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ branch-1 Compile Tests _ |
| +1 :green_heart: | mvninstall | 9m 42s | branch-1 passed |
| +1 :green_heart: | compile | 0m 39s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | compile | 0m 44s | branch-1 passed with JDK v1.7.0_272 |
| +1 :green_heart: | checkstyle | 1m 44s | branch-1 passed |
| +1 :green_heart: | shadedjars | 3m 1s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 48s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 41s | branch-1 passed with JDK v1.7.0_272 |
| +0 :ok: | spotbugs | 3m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 58s | branch-1 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 56s | the patch passed |
| +1 :green_heart: | compile | 0m 39s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javac | 0m 39s | the patch passed |
| +1 :green_heart: | compile | 0m 46s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | javac | 0m 46s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 31s | hbase-server: The patch generated 0 new + 44 unchanged - 4 fixed = 44 total (was 48) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 2m 49s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 4m 36s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. |
| +1 :green_heart: | javadoc | 0m 32s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 40s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | findbugs | 2m 57s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 122m 51s | hbase-server in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | The patch does not generate ASF License warnings. |
| | | 165m 3s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2591 |
| JIRA Issue | HBASE-24859 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d25d6a5431cc 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2591/out/precommit/personality/provided.sh |
| git revision | branch-1 / 8813b3b |
| Default Java | 1.7.0_272 |
| Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/7/testReport/ |
| Max. process+thread count | 3878 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/7/console |
| versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#discussion_r514590686
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
##########
@@ -271,6 +272,11 @@ public void testNumOfSplits(int splitsPerRegion, int expectedNumOfSplits) throws
TableInputFormat tif = new TableInputFormat();
tif.setConf(job.getConfiguration());
List<InputSplit> splits = tif.getSplits(job);
+ for (InputSplit split : splits) {
+ TableSplit tableSplit = (TableSplit) split;
+ Assert.assertEquals(tableSplit.getScan().getStartRow(), HConstants.EMPTY_START_ROW);
Review comment:
Remove the other two asserts? They are implied if the scan is "" ?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java
##########
@@ -86,7 +86,16 @@ static Version fromCode(int code) {
private byte [] endRow;
private String regionLocation;
private String encodedRegionName = "";
+
+ /** The scan object may be null but the serialized form of scan is never null
Review comment:
nit: new lines missing
```
/**
* .....
*/
```
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java
##########
@@ -214,6 +223,14 @@ public Scan getScan() throws IOException {
return TableMapReduceUtil.convertStringToScan(this.scan);
}
+ /**
+ * Returns a scan object in the serialized form
+ * @return a serialized scan object
+ */
+ public String getScanAsString() {
Review comment:
Mark this InterfaceAudience.Private? This is a test helper method..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#issuecomment-719165758
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ branch-1 Compile Tests _ |
| +1 :green_heart: | mvninstall | 9m 53s | branch-1 passed |
| +1 :green_heart: | compile | 0m 40s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | compile | 0m 44s | branch-1 passed with JDK v1.7.0_272 |
| +1 :green_heart: | checkstyle | 1m 43s | branch-1 passed |
| +1 :green_heart: | shadedjars | 3m 5s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 48s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 41s | branch-1 passed with JDK v1.7.0_272 |
| +0 :ok: | spotbugs | 3m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 1s | branch-1 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 55s | the patch passed |
| +1 :green_heart: | compile | 0m 41s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javac | 0m 41s | the patch passed |
| +1 :green_heart: | compile | 0m 45s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | javac | 0m 45s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 30s | hbase-server: The patch generated 0 new + 42 unchanged - 6 fixed = 42 total (was 48) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 2m 51s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 4m 34s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. |
| +1 :green_heart: | javadoc | 0m 32s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 42s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | findbugs | 2m 54s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 121m 51s | hbase-server in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | The patch does not generate ASF License warnings. |
| | | 164m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2591 |
| JIRA Issue | HBASE-24859 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7b60796fe67d 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2591/out/precommit/personality/provided.sh |
| git revision | branch-1 / 8813b3b |
| Default Java | 1.7.0_272 |
| Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/8/testReport/ |
| Max. process+thread count | 4143 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/8/console |
| versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2591: HBASE-24859: Optimize in-memory representation of HBase map reduce table splits
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2591:
URL: https://github.com/apache/hbase/pull/2591#issuecomment-718374329
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 18s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ branch-1 Compile Tests _ |
| +1 :green_heart: | mvninstall | 9m 52s | branch-1 passed |
| +1 :green_heart: | compile | 0m 41s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | compile | 0m 47s | branch-1 passed with JDK v1.7.0_272 |
| +1 :green_heart: | checkstyle | 1m 44s | branch-1 passed |
| +1 :green_heart: | shadedjars | 3m 5s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 50s | branch-1 passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 42s | branch-1 passed with JDK v1.7.0_272 |
| +0 :ok: | spotbugs | 3m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 59s | branch-1 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 59s | the patch passed |
| +1 :green_heart: | compile | 0m 38s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javac | 0m 38s | the patch passed |
| +1 :green_heart: | compile | 0m 44s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | javac | 0m 44s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 2m 47s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 4m 39s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. |
| +1 :green_heart: | javadoc | 0m 31s | the patch passed with JDK v1.8.0_262 |
| +1 :green_heart: | javadoc | 0m 39s | the patch passed with JDK v1.7.0_272 |
| +1 :green_heart: | findbugs | 2m 52s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 136m 42s | hbase-server in the patch failed. |
| +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
| | | 178m 59s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hbase.regionserver.TestSplitTransactionOnCluster |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2591 |
| JIRA Issue | HBASE-24859 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1238045de6ef 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2591/out/precommit/personality/provided.sh |
| git revision | branch-1 / b30d1d1 |
| Default Java | 1.7.0_272 |
| Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/5/artifact/out/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/5/testReport/ |
| Max. process+thread count | 4025 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2591/5/console |
| versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org