You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "hgromer (via GitHub)" <gi...@apache.org> on 2025/09/11 16:49:11 UTC
[PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
hgromer opened a new pull request, #7294:
URL: https://github.com/apache/hbase/pull/7294
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on code in PR #7294:
URL: https://github.com/apache/hbase/pull/7294#discussion_r2356256553
##########
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/KeyOnlyCellComparable.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.mapreduce;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.ExtendedCell;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.PrivateCellUtil;
+import org.apache.hadoop.io.WritableComparable;
+import org.apache.hadoop.io.WritableComparator;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class KeyOnlyCellComparable implements WritableComparable<KeyOnlyCellComparable> {
+
+ static {
+ WritableComparator.define(KeyOnlyCellComparable.class, new KeyOnlyCellComparator());
+ }
+
+ private ExtendedCell cell = null;
+
+ public KeyOnlyCellComparable() {
+ }
+
+ public KeyOnlyCellComparable(ExtendedCell cell) {
+ this.cell = cell;
+ }
+
+ public ExtendedCell getCell() {
+ return cell;
+ }
+
+ @Override
+ @edu.umd.cs.findbugs.annotations.SuppressWarnings(value = "EQ_COMPARETO_USE_OBJECT_EQUALS",
+ justification = "This is wrong, yes, but we should be purging Writables, not fixing them")
+ public int compareTo(KeyOnlyCellComparable o) {
+ return CellComparator.getInstance().compare(cell, o.cell);
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ int keyLen = PrivateCellUtil.estimatedSerializedSizeOfKey(cell);
+ int valueLen = 0; // We avoid writing value here. So just serialize as if an empty value.
+ out.writeInt(keyLen + valueLen + KeyValue.KEYVALUE_INFRASTRUCTURE_SIZE);
+ out.writeInt(keyLen);
+ out.writeInt(valueLen);
+ PrivateCellUtil.writeFlatKey(cell, out);
+ out.writeLong(cell.getSequenceId());
+ }
+
+ @Override
+ public void readFields(DataInput in) throws IOException {
+ cell = KeyValue.create(in);
+ long seqId = in.readLong();
+ cell.setSequenceId(seqId);
+ }
+
+ public static class KeyOnlyCellComparator extends WritableComparator {
+
+ @Override
+ public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
Review Comment:
I'm following prior art from CellWritableComparator, but let me see if that works, that would be a good optimization for sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "rmdmattingly (via GitHub)" <gi...@apache.org>.
rmdmattingly commented on code in PR #7294:
URL: https://github.com/apache/hbase/pull/7294#discussion_r2353625973
##########
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/PreSortedCellsReducer.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.mapreduce;
+
+import java.io.IOException;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.PrivateCellUtil;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.MapReduceExtendedCell;
+import org.apache.hadoop.mapreduce.Reducer;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class PreSortedCellsReducer
+ extends Reducer<KeyOnlyCellComparable, Cell, ImmutableBytesWritable, Cell> {
+
+ @Override
+ protected void reduce(KeyOnlyCellComparable key, Iterable<Cell> values, Context context)
+ throws IOException, InterruptedException {
+
+ int index = 0;
+ for (Cell cell : values) {
+ context.write(new ImmutableBytesWritable(CellUtil.cloneRow(key.getCell())),
+ new MapReduceExtendedCell(PrivateCellUtil.ensureExtendedCell(cell)));
Review Comment:
It's probably worth the optimization of only building this ImmutableBytesWritable once? Or is it a bug that we're calling key#getCell within the loop here?
##########
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/KeyOnlyCellComparable.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.mapreduce;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.ExtendedCell;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.PrivateCellUtil;
+import org.apache.hadoop.io.WritableComparable;
+import org.apache.hadoop.io.WritableComparator;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class KeyOnlyCellComparable implements WritableComparable<KeyOnlyCellComparable> {
+
+ static {
+ WritableComparator.define(KeyOnlyCellComparable.class, new KeyOnlyCellComparator());
+ }
+
+ private ExtendedCell cell = null;
+
+ public KeyOnlyCellComparable() {
+ }
+
+ public KeyOnlyCellComparable(ExtendedCell cell) {
+ this.cell = cell;
+ }
+
+ public ExtendedCell getCell() {
+ return cell;
+ }
+
+ @Override
+ @edu.umd.cs.findbugs.annotations.SuppressWarnings(value = "EQ_COMPARETO_USE_OBJECT_EQUALS",
+ justification = "This is wrong, yes, but we should be purging Writables, not fixing them")
+ public int compareTo(KeyOnlyCellComparable o) {
+ return CellComparator.getInstance().compare(cell, o.cell);
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ int keyLen = PrivateCellUtil.estimatedSerializedSizeOfKey(cell);
+ int valueLen = 0; // We avoid writing value here. So just serialize as if an empty value.
+ out.writeInt(keyLen + valueLen + KeyValue.KEYVALUE_INFRASTRUCTURE_SIZE);
+ out.writeInt(keyLen);
+ out.writeInt(valueLen);
+ PrivateCellUtil.writeFlatKey(cell, out);
+ out.writeLong(cell.getSequenceId());
+ }
+
+ @Override
+ public void readFields(DataInput in) throws IOException {
+ cell = KeyValue.create(in);
+ long seqId = in.readLong();
+ cell.setSequenceId(seqId);
+ }
+
+ public static class KeyOnlyCellComparator extends WritableComparator {
+
+ @Override
+ public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
Review Comment:
You might want to reuse the CellComparator here — it's probably better practice, and probably faster, vs doing so much full deserialization. Something like
```java
@Override
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
int kvLen1 = Bytes.toInt(b1, s1);
int keyLen1 = Bytes.toInt(b1, s1 + 4);
int kvLen2 = Bytes.toInt(b2, s2);
int keyLen2 = Bytes.toInt(b2, s2 + 4);
int keyOffset1 = s1 + 12; // 4 + 4 + 4 bytes
int keyOffset2 = s2 + 12;
return CellComparator.getInstance().compareKeys(
b1, keyOffset1, keyLen1,
b2, keyOffset2, keyLen2
);
}
```
##########
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/KeyOnlyCellComparable.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.mapreduce;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.ExtendedCell;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.PrivateCellUtil;
+import org.apache.hadoop.io.WritableComparable;
+import org.apache.hadoop.io.WritableComparator;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class KeyOnlyCellComparable implements WritableComparable<KeyOnlyCellComparable> {
+
+ static {
+ WritableComparator.define(KeyOnlyCellComparable.class, new KeyOnlyCellComparator());
+ }
+
+ private ExtendedCell cell = null;
+
+ public KeyOnlyCellComparable() {
+ }
+
+ public KeyOnlyCellComparable(ExtendedCell cell) {
+ this.cell = cell;
+ }
+
+ public ExtendedCell getCell() {
+ return cell;
+ }
+
+ @Override
+ @edu.umd.cs.findbugs.annotations.SuppressWarnings(value = "EQ_COMPARETO_USE_OBJECT_EQUALS",
+ justification = "This is wrong, yes, but we should be purging Writables, not fixing them")
+ public int compareTo(KeyOnlyCellComparable o) {
+ return CellComparator.getInstance().compare(cell, o.cell);
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ int keyLen = PrivateCellUtil.estimatedSerializedSizeOfKey(cell);
+ int valueLen = 0; // We avoid writing value here. So just serialize as if an empty value.
+ out.writeInt(keyLen + valueLen + KeyValue.KEYVALUE_INFRASTRUCTURE_SIZE);
+ out.writeInt(keyLen);
+ out.writeInt(valueLen);
+ PrivateCellUtil.writeFlatKey(cell, out);
+ out.writeLong(cell.getSequenceId());
+ }
+
+ @Override
+ public void readFields(DataInput in) throws IOException {
+ cell = KeyValue.create(in);
+ long seqId = in.readLong();
+ cell.setSequenceId(seqId);
+ }
+
+ public static class KeyOnlyCellComparator extends WritableComparator {
+
+ @Override
+ public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
+ try {
+ KeyOnlyCellComparable kv1 = new KeyOnlyCellComparable();
+ kv1.readFields(new DataInputStream(new ByteArrayInputStream(b1, s1, l1)));
+ KeyOnlyCellComparable kv2 = new KeyOnlyCellComparable();
+ kv2.readFields(new DataInputStream(new ByteArrayInputStream(b2, s2, l2)));
Review Comment:
Do we need to close these DataInputStreams?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3282353608
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 22s | | master passed |
| +1 :green_heart: | compile | 0m 39s | | master passed |
| +1 :green_heart: | javadoc | 0m 28s | | master passed |
| +1 :green_heart: | shadedjars | 6m 12s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 10s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 11s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 21m 2s | [/patch-unit-hbase-mapreduce.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/2/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-mapreduce.txt) | hbase-mapreduce in the patch failed. |
| +1 :green_heart: | unit | 11m 10s | | hbase-backup in the patch passed. |
| | | 55m 36s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux a2eaa31c2b50 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 007b42d9380e9a24af89dd0f7664a9922095456d |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/2/testReport/ |
| Max. process+thread count | 3673 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/2/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on code in PR #7294:
URL: https://github.com/apache/hbase/pull/7294#discussion_r2355376207
##########
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/KeyOnlyCellComparable.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.mapreduce;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.ExtendedCell;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.PrivateCellUtil;
+import org.apache.hadoop.io.WritableComparable;
+import org.apache.hadoop.io.WritableComparator;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class KeyOnlyCellComparable implements WritableComparable<KeyOnlyCellComparable> {
+
+ static {
+ WritableComparator.define(KeyOnlyCellComparable.class, new KeyOnlyCellComparator());
+ }
+
+ private ExtendedCell cell = null;
+
+ public KeyOnlyCellComparable() {
+ }
+
+ public KeyOnlyCellComparable(ExtendedCell cell) {
+ this.cell = cell;
+ }
+
+ public ExtendedCell getCell() {
+ return cell;
+ }
+
+ @Override
+ @edu.umd.cs.findbugs.annotations.SuppressWarnings(value = "EQ_COMPARETO_USE_OBJECT_EQUALS",
+ justification = "This is wrong, yes, but we should be purging Writables, not fixing them")
+ public int compareTo(KeyOnlyCellComparable o) {
+ return CellComparator.getInstance().compare(cell, o.cell);
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ int keyLen = PrivateCellUtil.estimatedSerializedSizeOfKey(cell);
+ int valueLen = 0; // We avoid writing value here. So just serialize as if an empty value.
+ out.writeInt(keyLen + valueLen + KeyValue.KEYVALUE_INFRASTRUCTURE_SIZE);
+ out.writeInt(keyLen);
+ out.writeInt(valueLen);
+ PrivateCellUtil.writeFlatKey(cell, out);
+ out.writeLong(cell.getSequenceId());
+ }
+
+ @Override
+ public void readFields(DataInput in) throws IOException {
+ cell = KeyValue.create(in);
+ long seqId = in.readLong();
+ cell.setSequenceId(seqId);
+ }
+
+ public static class KeyOnlyCellComparator extends WritableComparator {
+
+ @Override
+ public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
+ try {
+ KeyOnlyCellComparable kv1 = new KeyOnlyCellComparable();
+ kv1.readFields(new DataInputStream(new ByteArrayInputStream(b1, s1, l1)));
+ KeyOnlyCellComparable kv2 = new KeyOnlyCellComparable();
+ kv2.readFields(new DataInputStream(new ByteArrayInputStream(b2, s2, l2)));
Review Comment:
No the calling framework will manage the streams that are passed to this class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3282088860
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 17s | | master passed |
| +1 :green_heart: | compile | 0m 39s | | master passed |
| +1 :green_heart: | javadoc | 0m 28s | | master passed |
| +1 :green_heart: | shadedjars | 6m 12s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 10s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 10s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 21m 45s | [/patch-unit-hbase-mapreduce.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/1/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-mapreduce.txt) | hbase-mapreduce in the patch failed. |
| +1 :green_heart: | unit | 11m 9s | | hbase-backup in the patch passed. |
| | | 57m 38s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux d82cd53851a2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d142add94e3acb9526e14aabe9bea6dd26358177 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/1/testReport/ |
| Max. process+thread count | 3688 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3304225236
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 56s | | master passed |
| +1 :green_heart: | compile | 1m 59s | | master passed |
| +1 :green_heart: | checkstyle | 0m 37s | | master passed |
| +1 :green_heart: | spotbugs | 1m 46s | | master passed |
| +1 :green_heart: | spotless | 1m 14s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 36s | | the patch passed |
| +1 :green_heart: | javac | 1m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 39s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 55s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 15m 50s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 1m 9s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not generate ASF License warnings. |
| | | 48m 6s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/9/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux bafdc902714c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 5bf07f95679b3052292c3bd8b3902ffd7cab3b3b |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 83 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/9/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3285535871
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 8s | | master passed |
| +1 :green_heart: | compile | 0m 39s | | master passed |
| +1 :green_heart: | javadoc | 0m 27s | | master passed |
| +1 :green_heart: | shadedjars | 6m 11s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed |
| +1 :green_heart: | shadedjars | 7m 49s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 25m 6s | | hbase-mapreduce in the patch passed. |
| +1 :green_heart: | unit | 12m 3s | | hbase-backup in the patch passed. |
| | | 62m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/5/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 8a1fa931d260 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 14145f875e16bb530bdbd918515aecae28242e5d |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/5/testReport/ |
| Max. process+thread count | 3673 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/5/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3285189302
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 58s | | master passed |
| +1 :green_heart: | compile | 1m 30s | | master passed |
| +1 :green_heart: | checkstyle | 0m 28s | | master passed |
| +1 :green_heart: | spotbugs | 1m 12s | | master passed |
| +1 :green_heart: | spotless | 0m 55s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 0s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 26s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 30s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 13m 16s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 58s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not generate ASF License warnings. |
| | | 40m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 53b3fca00a40 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 38ff82848db035d93de5b30c23658346034bf216 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 81 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/4/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3286451445
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 18s | | master passed |
| +1 :green_heart: | compile | 1m 10s | | master passed |
| +1 :green_heart: | checkstyle | 0m 23s | | master passed |
| +1 :green_heart: | spotbugs | 1m 6s | | master passed |
| +1 :green_heart: | spotless | 0m 49s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 6s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 18s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 12m 30s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 51s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 19s | | The patch does not generate ASF License warnings. |
| | | 36m 12s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/7/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux c157d3b4d79a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9b9bd7c0a9986210a2cad3193af780f367099d53 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/7/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on code in PR #7294:
URL: https://github.com/apache/hbase/pull/7294#discussion_r2353656560
##########
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/PreSortedCellsReducer.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.mapreduce;
+
+import java.io.IOException;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.PrivateCellUtil;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.util.MapReduceExtendedCell;
+import org.apache.hadoop.mapreduce.Reducer;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class PreSortedCellsReducer
+ extends Reducer<KeyOnlyCellComparable, Cell, ImmutableBytesWritable, Cell> {
+
+ @Override
+ protected void reduce(KeyOnlyCellComparable key, Iterable<Cell> values, Context context)
+ throws IOException, InterruptedException {
+
+ int index = 0;
+ for (Cell cell : values) {
+ context.write(new ImmutableBytesWritable(CellUtil.cloneRow(key.getCell())),
+ new MapReduceExtendedCell(PrivateCellUtil.ensureExtendedCell(cell)));
Review Comment:
Ah no you're right, we can optimize here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3299136825
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 3m 13s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 36s | | master passed |
| +1 :green_heart: | compile | 1m 12s | | master passed |
| +1 :green_heart: | checkstyle | 0m 25s | | master passed |
| +1 :green_heart: | spotbugs | 1m 32s | | master passed |
| +1 :green_heart: | spotless | 0m 49s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 12s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 12m 17s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 45s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 17s | | The patch does not generate ASF License warnings. |
| | | 39m 46s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/8/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 1ac00171358d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ed46a5a9419c603672b8a024ae40cf6c1771df65 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/8/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on code in PR #7294:
URL: https://github.com/apache/hbase/pull/7294#discussion_r2356333077
##########
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/KeyOnlyCellComparable.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.mapreduce;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.ExtendedCell;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.PrivateCellUtil;
+import org.apache.hadoop.io.WritableComparable;
+import org.apache.hadoop.io.WritableComparator;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class KeyOnlyCellComparable implements WritableComparable<KeyOnlyCellComparable> {
+
+ static {
+ WritableComparator.define(KeyOnlyCellComparable.class, new KeyOnlyCellComparator());
+ }
+
+ private ExtendedCell cell = null;
+
+ public KeyOnlyCellComparable() {
+ }
+
+ public KeyOnlyCellComparable(ExtendedCell cell) {
+ this.cell = cell;
+ }
+
+ public ExtendedCell getCell() {
+ return cell;
+ }
+
+ @Override
+ @edu.umd.cs.findbugs.annotations.SuppressWarnings(value = "EQ_COMPARETO_USE_OBJECT_EQUALS",
+ justification = "This is wrong, yes, but we should be purging Writables, not fixing them")
+ public int compareTo(KeyOnlyCellComparable o) {
+ return CellComparator.getInstance().compare(cell, o.cell);
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ int keyLen = PrivateCellUtil.estimatedSerializedSizeOfKey(cell);
+ int valueLen = 0; // We avoid writing value here. So just serialize as if an empty value.
+ out.writeInt(keyLen + valueLen + KeyValue.KEYVALUE_INFRASTRUCTURE_SIZE);
+ out.writeInt(keyLen);
+ out.writeInt(valueLen);
+ PrivateCellUtil.writeFlatKey(cell, out);
+ out.writeLong(cell.getSequenceId());
+ }
+
+ @Override
+ public void readFields(DataInput in) throws IOException {
+ cell = KeyValue.create(in);
+ long seqId = in.readLong();
+ cell.setSequenceId(seqId);
+ }
+
+ public static class KeyOnlyCellComparator extends WritableComparator {
+
+ @Override
+ public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
Review Comment:
Fwiw, we do end up using CellComparator (compareTo) method, we're just using the compare(Cell a, Cell b) method. I don't think there's a good way do this cleanly without first creating the cells from what I can see
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3304299507
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 21s | | master passed |
| +1 :green_heart: | compile | 0m 53s | | master passed |
| +1 :green_heart: | javadoc | 0m 39s | | master passed |
| +1 :green_heart: | shadedjars | 7m 42s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 1s | | the patch passed |
| +1 :green_heart: | javac | 1m 1s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed |
| +1 :green_heart: | shadedjars | 7m 15s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 28m 27s | | hbase-mapreduce in the patch passed. |
| +1 :green_heart: | unit | 15m 14s | | hbase-backup in the patch passed. |
| | | 73m 10s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/9/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 4e7c85a86a5c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 5bf07f95679b3052292c3bd8b3902ffd7cab3b3b |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/9/testReport/ |
| Max. process+thread count | 3653 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/9/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on code in PR #7294:
URL: https://github.com/apache/hbase/pull/7294#discussion_r2355376207
##########
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/KeyOnlyCellComparable.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.mapreduce;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.ExtendedCell;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.PrivateCellUtil;
+import org.apache.hadoop.io.WritableComparable;
+import org.apache.hadoop.io.WritableComparator;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class KeyOnlyCellComparable implements WritableComparable<KeyOnlyCellComparable> {
+
+ static {
+ WritableComparator.define(KeyOnlyCellComparable.class, new KeyOnlyCellComparator());
+ }
+
+ private ExtendedCell cell = null;
+
+ public KeyOnlyCellComparable() {
+ }
+
+ public KeyOnlyCellComparable(ExtendedCell cell) {
+ this.cell = cell;
+ }
+
+ public ExtendedCell getCell() {
+ return cell;
+ }
+
+ @Override
+ @edu.umd.cs.findbugs.annotations.SuppressWarnings(value = "EQ_COMPARETO_USE_OBJECT_EQUALS",
+ justification = "This is wrong, yes, but we should be purging Writables, not fixing them")
+ public int compareTo(KeyOnlyCellComparable o) {
+ return CellComparator.getInstance().compare(cell, o.cell);
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ int keyLen = PrivateCellUtil.estimatedSerializedSizeOfKey(cell);
+ int valueLen = 0; // We avoid writing value here. So just serialize as if an empty value.
+ out.writeInt(keyLen + valueLen + KeyValue.KEYVALUE_INFRASTRUCTURE_SIZE);
+ out.writeInt(keyLen);
+ out.writeInt(valueLen);
+ PrivateCellUtil.writeFlatKey(cell, out);
+ out.writeLong(cell.getSequenceId());
+ }
+
+ @Override
+ public void readFields(DataInput in) throws IOException {
+ cell = KeyValue.create(in);
+ long seqId = in.readLong();
+ cell.setSequenceId(seqId);
+ }
+
+ public static class KeyOnlyCellComparator extends WritableComparator {
+
+ @Override
+ public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
+ try {
+ KeyOnlyCellComparable kv1 = new KeyOnlyCellComparable();
+ kv1.readFields(new DataInputStream(new ByteArrayInputStream(b1, s1, l1)));
+ KeyOnlyCellComparable kv2 = new KeyOnlyCellComparable();
+ kv2.readFields(new DataInputStream(new ByteArrayInputStream(b2, s2, l2)));
Review Comment:
No the calling framework will manage the streams that are passed to this class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on code in PR #7294:
URL: https://github.com/apache/hbase/pull/7294#discussion_r2349897007
##########
hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java:
##########
@@ -335,7 +351,29 @@ public void testMainMethod() throws Exception {
System.setErr(oldPrintStream);
System.setSecurityManager(SECURITY_MANAGER);
}
+ }
+
+ private static void runWithDiskBasedSortingDisabledAndEnabled(TestMethod method)
Review Comment:
Let me know if you disagree with this testing strategy, but I felt that the existing tests were comprehensive and this was a good way of running with both versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3282283805
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 11s | | master passed |
| +1 :green_heart: | compile | 1m 6s | | master passed |
| +1 :green_heart: | checkstyle | 0m 23s | | master passed |
| +1 :green_heart: | spotbugs | 0m 57s | | master passed |
| +1 :green_heart: | spotless | 0m 45s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 7s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 12s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 11m 57s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| -1 :x: | spotless | 0m 39s | | patch has 21 errors when running spotless:check, run spotless:apply to fix. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 17s | | The patch does not generate ASF License warnings. |
| | | 33m 48s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux ae08b24c8837 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 007b42d9380e9a24af89dd0f7664a9922095456d |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/2/artifact/yetus-general-check/output/patch-spotless.txt |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/2/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3286206460
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 9s | | master passed |
| +1 :green_heart: | compile | 0m 40s | | master passed |
| +1 :green_heart: | javadoc | 0m 29s | | master passed |
| +1 :green_heart: | shadedjars | 6m 8s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 3s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 4s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 21m 42s | | hbase-mapreduce in the patch passed. |
| +1 :green_heart: | unit | 11m 6s | | hbase-backup in the patch passed. |
| | | 55m 49s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/6/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 1feebdb44746 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / c025beb4fe11ffb1e287d2d1172068650a962f01 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/6/testReport/ |
| Max. process+thread count | 3659 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/6/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3285447807
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 45s | | master passed |
| +1 :green_heart: | compile | 1m 27s | | master passed |
| +1 :green_heart: | checkstyle | 0m 28s | | master passed |
| +1 :green_heart: | spotbugs | 1m 9s | | master passed |
| +1 :green_heart: | spotless | 0m 52s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 4s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 28s | | the patch passed |
| -1 :x: | spotbugs | 0m 51s | [/new-spotbugs-hbase-mapreduce.html](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/5/artifact/yetus-general-check/output/new-spotbugs-hbase-mapreduce.html) | hbase-mapreduce generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | hadoopcheck | 13m 9s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 49s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 17s | | The patch does not generate ASF License warnings. |
| | | 38m 57s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hbase-mapreduce |
| | org.apache.hadoop.hbase.mapreduce.KeyOnlyCellComparable defines compareTo(KeyOnlyCellComparable) and uses Object.equals() At KeyOnlyCellComparable.java:Object.equals() At KeyOnlyCellComparable.java:[line 55] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 3b2c02fdb621 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 14145f875e16bb530bdbd918515aecae28242e5d |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 83 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/5/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3299207842
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 10s | | master passed |
| +1 :green_heart: | compile | 0m 40s | | master passed |
| +1 :green_heart: | javadoc | 0m 31s | | master passed |
| +1 :green_heart: | shadedjars | 6m 8s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 1s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 4s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 21m 28s | | hbase-mapreduce in the patch passed. |
| +1 :green_heart: | unit | 11m 5s | | hbase-backup in the patch passed. |
| | | 55m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/8/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 882b2e727ab1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ed46a5a9419c603672b8a024ae40cf6c1771df65 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/8/testReport/ |
| Max. process+thread count | 3673 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/8/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3282567794
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 9s | | master passed |
| +1 :green_heart: | compile | 0m 40s | | master passed |
| +1 :green_heart: | javadoc | 0m 27s | | master passed |
| +1 :green_heart: | shadedjars | 6m 16s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 9s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 11s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 21m 24s | [/patch-unit-hbase-mapreduce.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/3/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-mapreduce.txt) | hbase-mapreduce in the patch failed. |
| +1 :green_heart: | unit | 11m 3s | | hbase-backup in the patch passed. |
| | | 55m 41s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 560293c910ad 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / dce93e4641061735c4e7763366fdf355e1399293 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/3/testReport/ |
| Max. process+thread count | 3662 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/3/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on code in PR #7294:
URL: https://github.com/apache/hbase/pull/7294#discussion_r2352313536
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java:
##########
@@ -421,6 +425,7 @@ protected void walToHFiles(List<String> dirPaths, List<String> tableList) throws
if (result != 0) {
throw new IOException("WAL Player failed");
}
+ conf.unset(HFileOutputFormat2.DISK_BASED_SORTING_ENABLED_KEY);
Review Comment:
Makes sense, wonder if it also makes sense for the other values as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3285246310
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 6s | | master passed |
| +1 :green_heart: | compile | 0m 39s | | master passed |
| +1 :green_heart: | javadoc | 0m 27s | | master passed |
| +1 :green_heart: | shadedjars | 6m 13s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 9s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 11s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 21m 26s | | hbase-mapreduce in the patch passed. |
| +1 :green_heart: | unit | 11m 2s | | hbase-backup in the patch passed. |
| | | 55m 54s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/4/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 09d617c34913 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 38ff82848db035d93de5b30c23658346034bf216 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/4/testReport/ |
| Max. process+thread count | 3683 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/4/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3282507476
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 17s | | master passed |
| +1 :green_heart: | compile | 1m 6s | | master passed |
| +1 :green_heart: | checkstyle | 0m 22s | | master passed |
| +1 :green_heart: | spotbugs | 1m 1s | | master passed |
| +1 :green_heart: | spotless | 0m 49s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 8s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 13s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 12m 29s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 47s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 18s | | The patch does not generate ASF License warnings. |
| | | 34m 59s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux b00bc01d23b7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / dce93e4641061735c4e7763366fdf355e1399293 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/3/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3293586673
cc @ndimiduk @charlesconnell @DieterDP-ng @rmdmattingly @ndimiduk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "DieterDP-ng (via GitHub)" <gi...@apache.org>.
DieterDP-ng commented on code in PR #7294:
URL: https://github.com/apache/hbase/pull/7294#discussion_r2351557063
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java:
##########
@@ -421,6 +425,7 @@ protected void walToHFiles(List<String> dirPaths, List<String> tableList) throws
if (result != 0) {
throw new IOException("WAL Player failed");
}
+ conf.unset(HFileOutputFormat2.DISK_BASED_SORTING_ENABLED_KEY);
Review Comment:
Shouldn't we check and re-set the original value, rather than just unsetting here?
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java:
##########
@@ -360,6 +362,7 @@ protected void incrementalCopyHFiles(String[] files, String backupDest) throws I
+ " finished.");
} finally {
deleteBulkLoadDirectory();
+ conf.unset(HFileOutputFormat2.DISK_BASED_SORTING_ENABLED_KEY);
Review Comment:
Shouldn't we check and re-set the original value, rather than just unsetting here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on code in PR #7294:
URL: https://github.com/apache/hbase/pull/7294#discussion_r2356320462
##########
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/KeyOnlyCellComparable.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.mapreduce;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.ExtendedCell;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.PrivateCellUtil;
+import org.apache.hadoop.io.WritableComparable;
+import org.apache.hadoop.io.WritableComparator;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class KeyOnlyCellComparable implements WritableComparable<KeyOnlyCellComparable> {
+
+ static {
+ WritableComparator.define(KeyOnlyCellComparable.class, new KeyOnlyCellComparator());
+ }
+
+ private ExtendedCell cell = null;
+
+ public KeyOnlyCellComparable() {
+ }
+
+ public KeyOnlyCellComparable(ExtendedCell cell) {
+ this.cell = cell;
+ }
+
+ public ExtendedCell getCell() {
+ return cell;
+ }
+
+ @Override
+ @edu.umd.cs.findbugs.annotations.SuppressWarnings(value = "EQ_COMPARETO_USE_OBJECT_EQUALS",
+ justification = "This is wrong, yes, but we should be purging Writables, not fixing them")
+ public int compareTo(KeyOnlyCellComparable o) {
+ return CellComparator.getInstance().compare(cell, o.cell);
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ int keyLen = PrivateCellUtil.estimatedSerializedSizeOfKey(cell);
+ int valueLen = 0; // We avoid writing value here. So just serialize as if an empty value.
+ out.writeInt(keyLen + valueLen + KeyValue.KEYVALUE_INFRASTRUCTURE_SIZE);
+ out.writeInt(keyLen);
+ out.writeInt(valueLen);
+ PrivateCellUtil.writeFlatKey(cell, out);
+ out.writeLong(cell.getSequenceId());
+ }
+
+ @Override
+ public void readFields(DataInput in) throws IOException {
+ cell = KeyValue.create(in);
+ long seqId = in.readLong();
+ cell.setSequenceId(seqId);
+ }
+
+ public static class KeyOnlyCellComparator extends WritableComparator {
+
+ @Override
+ public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
+ try {
+ KeyOnlyCellComparable kv1 = new KeyOnlyCellComparable();
+ kv1.readFields(new DataInputStream(new ByteArrayInputStream(b1, s1, l1)));
+ KeyOnlyCellComparable kv2 = new KeyOnlyCellComparable();
+ kv2.readFields(new DataInputStream(new ByteArrayInputStream(b2, s2, l2)));
Review Comment:
Good catch yeah, can wrap them in the try()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "rmdmattingly (via GitHub)" <gi...@apache.org>.
rmdmattingly merged PR #7294:
URL: https://github.com/apache/hbase/pull/7294
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3286072011
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 10s | | master passed |
| +1 :green_heart: | compile | 1m 5s | | master passed |
| +1 :green_heart: | checkstyle | 0m 21s | | master passed |
| +1 :green_heart: | spotbugs | 0m 58s | | master passed |
| +1 :green_heart: | spotless | 0m 49s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 8s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 12s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 12m 8s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| -1 :x: | spotless | 0m 38s | | patch has 22 errors when running spotless:check, run spotless:apply to fix. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 17s | | The patch does not generate ASF License warnings. |
| | | 34m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux d97433e5bb49 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / c025beb4fe11ffb1e287d2d1172068650a962f01 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/6/artifact/yetus-general-check/output/patch-spotless.txt |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/6/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3286493899
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 21s | | master passed |
| +1 :green_heart: | compile | 0m 39s | | master passed |
| +1 :green_heart: | javadoc | 0m 29s | | master passed |
| +1 :green_heart: | shadedjars | 6m 15s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 13s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 12s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 28s | | hbase-mapreduce in the patch passed. |
| +1 :green_heart: | unit | 10m 37s | | hbase-backup in the patch passed. |
| | | 51m 48s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/7/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux d8ce44ac53ee 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9b9bd7c0a9986210a2cad3193af780f367099d53 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/7/testReport/ |
| Max. process+thread count | 3239 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/7/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7294:
URL: https://github.com/apache/hbase/pull/7294#issuecomment-3282018833
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 18s | | master passed |
| +1 :green_heart: | compile | 1m 11s | | master passed |
| +1 :green_heart: | checkstyle | 0m 21s | | master passed |
| +1 :green_heart: | spotbugs | 0m 58s | | master passed |
| +1 :green_heart: | spotless | 0m 50s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 7s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 12s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 12m 10s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 46s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 18s | | The patch does not generate ASF License warnings. |
| | | 34m 22s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7294 |
| JIRA Issue | HBASE-28440 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux e9af2ea86625 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d142add94e3acb9526e14aabe9bea6dd26358177 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 83 (vs. ulimit of 30000) |
| modules | C: hbase-mapreduce hbase-backup U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7294/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on code in PR #7294:
URL: https://github.com/apache/hbase/pull/7294#discussion_r2352319579
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java:
##########
@@ -421,13 +429,16 @@ protected void walToHFiles(List<String> dirPaths, List<String> tableList) throws
if (result != 0) {
throw new IOException("WAL Player failed");
}
- conf.unset(WALPlayer.INPUT_FILES_SEPARATOR_KEY);
- conf.unset(JOB_NAME_CONF_KEY);
} catch (IOException e) {
throw e;
} catch (Exception ee) {
throw new IOException("Can not convert from directory " + dirs
+ " (check Hadoop, HBase and WALPlayer M/R job logs) ", ee);
+ } finally {
+ conf.setBoolean(HFileOutputFormat2.DISK_BASED_SORTING_ENABLED_KEY,
+ diskBasedSortingEnabledOriginalValue);
+ conf.unset(WALPlayer.INPUT_FILES_SEPARATOR_KEY);
Review Comment:
Moving all of these into the `finally` block, otherwise we may not unset them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28440: Add support for using mapreduce sort in HFileOutputFormat2 [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on code in PR #7294:
URL: https://github.com/apache/hbase/pull/7294#discussion_r2352311815
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java:
##########
@@ -360,6 +362,7 @@ protected void incrementalCopyHFiles(String[] files, String backupDest) throws I
+ " finished.");
} finally {
deleteBulkLoadDirectory();
+ conf.unset(HFileOutputFormat2.DISK_BASED_SORTING_ENABLED_KEY);
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org