You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "swamirishi (via GitHub)" <gi...@apache.org> on 2023/04/18 09:13:12 UTC
[GitHub] [ozone] swamirishi opened a new pull request, #4584: HDDS-8306. [Snapshot] Add tests for SST dump tool
swamirishi opened a new pull request, #4584:
URL: https://github.com/apache/ozone/pull/4584
## What changes were proposed in this pull request?
Adding unit tests for SST Dump Tool. The test cases depends on #4470 .
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8306
## How was this patch tested?
Unit Tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Add tests for SST dump tool
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1183133786
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,109 @@
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.IntStream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ Map<Pair<String, Integer>, String> records)
+ throws Exception {
+ Map<Pair<String, Integer>, String> keys = records instanceof TreeMap ?
Review Comment:
1. These tests could be changed in to parameterized test.
2. If you expect `records` to be an instance of `TreeMap`, wouldn't it be better for requested to send a `TreeMap` itself.
3. I don't get (or like) idea of using `Pair<String, Integer>` as key. It could be simplified if you use parameterized tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191803314
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.ArgumentsProvider;
+import org.junit.jupiter.params.provider.ArgumentsSource;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191130888
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,109 @@
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.IntStream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ Map<Pair<String, Integer>, String> records)
+ throws Exception {
+ Map<Pair<String, Integer>, String> keys = records instanceof TreeMap ?
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195728768
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/NativeLibraryLoader.java:
##########
@@ -132,15 +134,17 @@ private Optional<File> copyResourceFromJarToTemp(final String libraryName)
}
// create a temporary file to copy the library to
- final File temp = File.createTempFile(libraryName, getLibOsSuffix());
+ final File temp = File.createTempFile(libraryName, getLibOsSuffix(),
+ new File(""));
Review Comment:
Hmm. Was extracting lib to temp dir causing problem because the temp would be cleared periodically?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195826123
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/NativeLibraryLoader.java:
##########
@@ -132,15 +134,17 @@ private Optional<File> copyResourceFromJarToTemp(final String libraryName)
}
// create a temporary file to copy the library to
- final File temp = File.createTempFile(libraryName, getLibOsSuffix());
+ final File temp = File.createTempFile(libraryName, getLibOsSuffix(),
+ new File(""));
Review Comment:
The temp could be cleared, that is why it is better to create in cwd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191766349
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/PipeInputStream.java:
##########
@@ -65,11 +65,13 @@ public int read() {
this.close();
return -1;
}
- if (numberOfBytesLeftToRead == 0) {
+ while (numberOfBytesLeftToRead == 0) {
numberOfBytesLeftToRead = readInternal(byteBuffer, capacity,
nativeHandle);
index = 0;
- return read();
+ if (numberOfBytesLeftToRead != 0) {
Review Comment:
it can also return 0 bytes read if pipe has no data it is non blocking, I saw a failure occur like this in a unit test run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195514120
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/PipeInputStream.java:
##########
@@ -65,11 +65,13 @@ public int read() {
this.close();
return -1;
}
- if (numberOfBytesLeftToRead == 0) {
+ while (numberOfBytesLeftToRead == 0) {
numberOfBytesLeftToRead = readInternal(byteBuffer, capacity,
nativeHandle);
index = 0;
- return read();
+ if (numberOfBytesLeftToRead != 0) {
Review Comment:
it will return a -1 if it has reached the end of stream, it will return a 0 if the pipe is empty but the pipe is not closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195744685
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -86,51 +81,58 @@ public ManagedSSTDumpIterator(ManagedSSTDumpTool sstDumpTool,
* @return Optional of the integer empty if no integer exists
*/
private Optional<Integer> getNextNumberInStream() throws IOException {
- StringBuilder value = new StringBuilder();
- int val;
- while ((val = processOutput.read()) != -1) {
- if (val >= '0' && val <= '9') {
- value.append((char) val);
- } else if (value.length() > 0) {
- break;
- }
+ int n = processOutput.read(intBuffer, 0, 4);
+ if (n == 4) {
+ return Optional.of(ByteBuffer.wrap(intBuffer).getInt());
+ } else if (n >= 0) {
+ throw new IllegalStateException(String.format("Integer expects " +
+ "4 bytes to be read from the stream, but read only %d bytes", n));
}
- return value.length() > 0 ? Optional.of(Integer.valueOf(value.toString()))
- : Optional.empty();
+ return Optional.empty();
}
- /**
- * Reads the next n chars from the stream & makes a string.
- *
- * @param numberOfChars
- * @return String of next chars read
- * @throws IOException
- */
- private String readNextNumberOfCharsFromStream(int numberOfChars)
- throws IOException {
- StringBuilder value = new StringBuilder();
- while (numberOfChars > 0) {
- int noOfCharsRead = processOutput.read(charBuffer, 0,
- Math.min(numberOfChars, charBuffer.length));
- if (noOfCharsRead == -1) {
- break;
+ private Optional<byte[]> getNextByteArray() throws IOException {
+ Optional<Integer> size = getNextNumberInStream();
+ if (size.isPresent()) {
+ byte[] b = new byte[size.get()];
Review Comment:
debug logging the byte array size
```suggestion
if (size.isPresent()) {
if (LOG.isDebugEnabled()) {
LOG.debug("Allocating byte array, size: {}", size.get());
}
byte[] b = new byte[size.get()];
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Add tests for SST dump tool
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191539110
##########
hadoop-hdds/rocks-native/pom.xml:
##########
@@ -33,6 +33,31 @@
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-io</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.github.spotbugs</groupId>
+ <artifactId>spotbugs-annotations</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-api</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-params</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
Review Comment:
Why do you need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191803508
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.ArgumentsProvider;
+import org.junit.jupiter.params.provider.ArgumentsSource;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue> iterator =
+ new ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue>(
+ tool, file.getAbsolutePath(), new ManagedOptions()) {
+
+ @Override
+ protected KeyValue getTransformedValue(Optional<KeyValue> value) {
+ return value.orElse(null);
+ }
+ }) {
+ while (iterator.hasNext()) {
+ ManagedSSTDumpIterator.KeyValue r = iterator.next();
+ Pair<String, Integer> recordKey = Pair.of(new String(r.getKey(),
+ StandardCharsets.UTF_8), r.getType());
+ Assertions.assertTrue(keys.containsKey(recordKey));
+ Assertions.assertEquals(
+ Optional.ofNullable(keys.get(recordKey)).orElse(""),
+ new String(r.getValue(), StandardCharsets.UTF_8));
+ keys.remove(recordKey);
+ }
+ Assertions.assertEquals(0, keys.size());
+ }
+ executorService.shutdown();
+ }
+ }
+
+ @Native("Managed Rocks Tools")
+ @ParameterizedTest
+ @ArgumentsSource(KeyValueFormatArgumentProvider.class)
+ public void testSSTDumpIteratorWithKeyFormat(String keyFormat,
+ String valueFormat) throws Exception {
+ TreeMap<Pair<String, Integer>, String> keys =
+ IntStream.range(0, 100).boxed().collect(
+ Collectors.toMap(
+ i -> Pair.of(String.format(keyFormat, i), i % 2),
+ i -> i % 2 == 0 ? "" : String.format(valueFormat, i),
+ (v1, v2) -> v2,
+ TreeMap::new));
+ testSSTDumpIteratorWithKeys(keys);
+ }
+}
+
+class KeyValueFormatArgumentProvider implements ArgumentsProvider {
+ @Override
+ public Stream<? extends Arguments> provideArguments(
+ ExtensionContext context) {
+ return Stream.of(
+ Arguments.of("'key%1$d=>", "%1$dvalue'"),
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191132443
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,109 @@
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.IntStream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ Map<Pair<String, Integer>, String> records)
+ throws Exception {
+ Map<Pair<String, Integer>, String> keys = records instanceof TreeMap ?
+ records : new TreeMap<>(records);
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ new ManagedEnvOptions(), new ManagedOptions())) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy()), 8192);
+ ManagedSSTDumpIterator iterator = new ManagedSSTDumpIterator(tool,
+ file.getAbsolutePath(), new ManagedOptions());
+ while (iterator.hasNext()) {
+ ManagedSSTDumpIterator.KeyValue r = iterator.next();
+ Pair<String, Integer> recordKey = Pair.of(r.getKey(), r.getType());
+ Assert.assertTrue(keys.containsKey(recordKey));
+ Assert.assertEquals(Optional.ofNullable(keys.get(recordKey)).orElse(""),
+ r.getValue());
+ keys.remove(recordKey);
+ }
+ Assert.assertEquals(keys.size(), 0);
+ iterator.close();
+ }
+ }
+ @Test
Review Comment:
done
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/NativeTests.java:
##########
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191856095
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -86,51 +79,47 @@ public ManagedSSTDumpIterator(ManagedSSTDumpTool sstDumpTool,
* @return Optional of the integer empty if no integer exists
*/
private Optional<Integer> getNextNumberInStream() throws IOException {
- StringBuilder value = new StringBuilder();
- int val;
- while ((val = processOutput.read()) != -1) {
- if (val >= '0' && val <= '9') {
- value.append((char) val);
- } else if (value.length() > 0) {
- break;
- }
+ int n = processOutput.read(intBuffer, 0, 4);
+ if (n == 4) {
Review Comment:
Done. Have added a test case for this as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191766587
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/NativeLibraryLoader.java:
##########
@@ -132,15 +134,17 @@ private Optional<File> copyResourceFromJarToTemp(final String libraryName)
}
// create a temporary file to copy the library to
- final File temp = File.createTempFile(libraryName, getLibOsSuffix());
+ final File temp = File.createTempFile(libraryName, getLibOsSuffix(),
+ new File(""));
Review Comment:
empty means current directory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195451457
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -244,40 +245,46 @@ protected void finalize() throws Throwable {
* Class containing Parsed KeyValue Record from Sst Dumptool output.
*/
public static final class KeyValue {
- private String key;
- private Integer sequence;
- private Integer type;
- private String value;
+ private final byte[] key;
+ private final UnsignedLong sequence;
+ private final Integer type;
+ private final byte[] value;
- private KeyValue(String key, String sequence, String type,
- String value) {
+ private KeyValue(byte[] key, UnsignedLong sequence, Integer type,
+ byte[] value) {
this.key = key;
- this.sequence = Integer.valueOf(sequence);
- this.type = Integer.valueOf(type);
+ this.sequence = sequence;
+ this.type = type;
this.value = value;
}
- public String getKey() {
+ @SuppressFBWarnings("EI_EXPOSE_REP")
+ public byte[] getKey() {
return key;
}
- public Integer getSequence() {
+ public UnsignedLong getSequence() {
return sequence;
}
public Integer getType() {
return type;
}
- public String getValue() {
+ @SuppressFBWarnings("EI_EXPOSE_REP")
+ public byte[] getValue() {
return value;
}
@Override
public String toString() {
- return "KeyValue{" + "key='" + key + '\'' + ", sequence=" + sequence +
- ", type=" + type + ", value='" + value + '\'' + '}';
+ return "KeyValue{" +
+ "key=" + StringUtils.bytes2String(key) +
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde merged pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "prashantpogde (via GitHub)" <gi...@apache.org>.
prashantpogde merged PR #4584:
URL: https://github.com/apache/ozone/pull/4584
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195746725
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -180,33 +182,33 @@ public boolean hasNext() {
public T next() {
checkSanityOfProcess();
currentKey = nextKey;
- nextKey = null;
- boolean keyFound = false;
- while (!keyFound) {
- try {
- Optional<Integer> keyLength = getNextNumberInStream();
- if (!keyLength.isPresent()) {
- return getTransformedValue(currentKey);
- }
- String keyStr = readNextNumberOfCharsFromStream(keyLength.get());
- Matcher matcher = PATTERN_MATCHER.matcher(keyStr);
- if (keyStr.length() == keyLength.get() && matcher.find()) {
- Optional<Integer> valueLength = getNextNumberInStream();
- if (valueLength.isPresent()) {
- String valueStr = readNextNumberOfCharsFromStream(
- valueLength.get());
- if (valueStr.length() == valueLength.get()) {
- keyFound = true;
- nextKey = new KeyValue(matcher.group(PATTERN_KEY_GROUP_NUMBER),
- matcher.group(PATTERN_SEQ_GROUP_NUMBER),
- matcher.group(PATTERN_TYPE_GROUP_NUMBER),
- valueStr);
- }
- }
- }
- } catch (IOException e) {
- throw new RuntimeIOException(e);
+ nextKey = Optional.empty();
+ try {
+ Optional<byte[]> key = getNextByteArray();
+ if (!key.isPresent()) {
+ return getTransformedValue(currentKey);
}
+ UnsignedLong sequenceNumber = getNextUnsignedLong()
+ .orElseThrow(() -> new IllegalStateException(
+ String.format("Error while trying to read sequence number" +
+ " for key %s", StringUtils.bytes2String(key.get()))));
+
+ Integer type = getNextNumberInStream()
+ .orElseThrow(() -> new IllegalStateException(
+ String.format("Error while trying to read sequence number for " +
+ "key %s with sequence number %s",
+ StringUtils.bytes2String(key.get()),
+ sequenceNumber.toString())));
+ byte[] val = getNextByteArray().orElseThrow(() ->
+ new IllegalStateException(
+ String.format("Error while trying to read sequence number for " +
+ "key %s with sequence number %s of type %d",
+ StringUtils.bytes2String(key.get()),
+ sequenceNumber.toString(), type)));
+ nextKey = Optional.of(new KeyValue(key.get(), sequenceNumber, type, val));
+ } catch (IOException e) {
+ // TODO [Snapshot] : Throw custom snapshot exception
Review Comment:
nit
```suggestion
// TODO: [SNAPSHOT] Throw custom snapshot exception
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1184272357
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,109 @@
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.IntStream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ Map<Pair<String, Integer>, String> records)
+ throws Exception {
+ Map<Pair<String, Integer>, String> keys = records instanceof TreeMap ?
+ records : new TreeMap<>(records);
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ new ManagedEnvOptions(), new ManagedOptions())) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy()), 8192);
+ ManagedSSTDumpIterator iterator = new ManagedSSTDumpIterator(tool,
+ file.getAbsolutePath(), new ManagedOptions());
+ while (iterator.hasNext()) {
+ ManagedSSTDumpIterator.KeyValue r = iterator.next();
+ Pair<String, Integer> recordKey = Pair.of(r.getKey(), r.getType());
+ Assert.assertTrue(keys.containsKey(recordKey));
+ Assert.assertEquals(Optional.ofNullable(keys.get(recordKey)).orElse(""),
+ r.getValue());
+ keys.remove(recordKey);
+ }
+ Assert.assertEquals(keys.size(), 0);
+ iterator.close();
+ }
+ }
+ @Test
Review Comment:
nit
```suggestion
}
@Test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1190984173
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191855710
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -86,51 +79,47 @@ public ManagedSSTDumpIterator(ManagedSSTDumpTool sstDumpTool,
* @return Optional of the integer empty if no integer exists
*/
private Optional<Integer> getNextNumberInStream() throws IOException {
- StringBuilder value = new StringBuilder();
- int val;
- while ((val = processOutput.read()) != -1) {
- if (val >= '0' && val <= '9') {
- value.append((char) val);
- } else if (value.length() > 0) {
- break;
- }
+ int n = processOutput.read(intBuffer, 0, 4);
+ if (n == 4) {
+ return Optional.of(ByteBuffer.wrap(intBuffer).getInt());
}
- return value.length() > 0 ? Optional.of(Integer.valueOf(value.toString()))
- : Optional.empty();
+ return Optional.empty();
}
- /**
- * Reads the next n chars from the stream & makes a string.
- *
- * @param numberOfChars
- * @return String of next chars read
- * @throws IOException
- */
- private String readNextNumberOfCharsFromStream(int numberOfChars)
- throws IOException {
- StringBuilder value = new StringBuilder();
- while (numberOfChars > 0) {
- int noOfCharsRead = processOutput.read(charBuffer, 0,
- Math.min(numberOfChars, charBuffer.length));
- if (noOfCharsRead == -1) {
- break;
- }
- value.append(charBuffer, 0, noOfCharsRead);
- numberOfChars -= noOfCharsRead;
+ private Optional<byte[]> getNextByteArray() throws IOException {
+ Optional<Integer> size = getNextNumberInStream();
+ if (size.isPresent()) {
+ byte[] b = new byte[size.get()];
+ int n = processOutput.read(b);
+ return n != size.get() ? Optional.empty() : Optional.of(b);
}
+ return Optional.empty();
+ }
- return value.toString();
+ private Optional<UnsignedLong> getNextUnsignedLong() {
+ long val = 0;
+ for (int i = 0; i < 8; i++) {
+ val = val << 8;
+ int nextByte = processOutput.read();
+ if (nextByte < 0) {
+ return Optional.empty();
+ }
+ val += nextByte;
+ }
+ return Optional.of(UnsignedLong.fromLongBits(val));
}
private void init(ManagedSSTDumpTool sstDumpTool, File sstFile,
ManagedOptions options)
throws NativeLibraryNotLoadedException {
- String[] args = {"--file=" + sstFile.getAbsolutePath(), "--command=scan"};
+ String[] args = {"--file=" + sstFile.getAbsolutePath(), "--command=scan",
Review Comment:
done
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -180,33 +169,25 @@ public boolean hasNext() {
public T next() {
checkSanityOfProcess();
currentKey = nextKey;
- nextKey = null;
- boolean keyFound = false;
- while (!keyFound) {
- try {
- Optional<Integer> keyLength = getNextNumberInStream();
- if (!keyLength.isPresent()) {
- return getTransformedValue(currentKey);
- }
- String keyStr = readNextNumberOfCharsFromStream(keyLength.get());
- Matcher matcher = PATTERN_MATCHER.matcher(keyStr);
- if (keyStr.length() == keyLength.get() && matcher.find()) {
- Optional<Integer> valueLength = getNextNumberInStream();
- if (valueLength.isPresent()) {
- String valueStr = readNextNumberOfCharsFromStream(
- valueLength.get());
- if (valueStr.length() == valueLength.get()) {
- keyFound = true;
- nextKey = new KeyValue(matcher.group(PATTERN_KEY_GROUP_NUMBER),
- matcher.group(PATTERN_SEQ_GROUP_NUMBER),
- matcher.group(PATTERN_TYPE_GROUP_NUMBER),
- valueStr);
+ nextKey = Optional.empty();
+ try {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191855104
##########
hadoop-hdds/rocks-native/pom.xml:
##########
@@ -33,6 +33,31 @@
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-io</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.github.spotbugs</groupId>
+ <artifactId>spotbugs-annotations</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-api</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-params</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195451157
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1192684061
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -180,33 +182,32 @@ public boolean hasNext() {
public T next() {
checkSanityOfProcess();
currentKey = nextKey;
- nextKey = null;
- boolean keyFound = false;
- while (!keyFound) {
- try {
- Optional<Integer> keyLength = getNextNumberInStream();
- if (!keyLength.isPresent()) {
- return getTransformedValue(currentKey);
- }
- String keyStr = readNextNumberOfCharsFromStream(keyLength.get());
- Matcher matcher = PATTERN_MATCHER.matcher(keyStr);
- if (keyStr.length() == keyLength.get() && matcher.find()) {
- Optional<Integer> valueLength = getNextNumberInStream();
- if (valueLength.isPresent()) {
- String valueStr = readNextNumberOfCharsFromStream(
- valueLength.get());
- if (valueStr.length() == valueLength.get()) {
- keyFound = true;
- nextKey = new KeyValue(matcher.group(PATTERN_KEY_GROUP_NUMBER),
- matcher.group(PATTERN_SEQ_GROUP_NUMBER),
- matcher.group(PATTERN_TYPE_GROUP_NUMBER),
- valueStr);
- }
- }
- }
- } catch (IOException e) {
- throw new RuntimeIOException(e);
+ nextKey = Optional.empty();
+ try {
+ Optional<byte[]> key = getNextByteArray();
+ if (!key.isPresent()) {
+ return getTransformedValue(currentKey);
}
+ UnsignedLong sequenceNumber = getNextUnsignedLong()
Review Comment:
getNextUnsignedLong() will throw an exception if it has number of bytes > 0. Here we are throwing an exception if the input stream has reached the end of the stream
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191762773
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -86,51 +79,47 @@ public ManagedSSTDumpIterator(ManagedSSTDumpTool sstDumpTool,
* @return Optional of the integer empty if no integer exists
*/
private Optional<Integer> getNextNumberInStream() throws IOException {
- StringBuilder value = new StringBuilder();
- int val;
- while ((val = processOutput.read()) != -1) {
- if (val >= '0' && val <= '9') {
- value.append((char) val);
- } else if (value.length() > 0) {
- break;
- }
+ int n = processOutput.read(intBuffer, 0, 4);
+ if (n == 4) {
Review Comment:
If that's the case, then we should throw an`IllegalStateException` if it happens and fail the request instead of returning `Optional.empty();`. I think we can keep returning `Optional.empty();` when there is no more byte (n is either 0 or -1) to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Add tests for SST dump tool
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191530724
##########
hadoop-hdds/rocks-native/pom.xml:
##########
@@ -33,6 +33,31 @@
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-io</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.github.spotbugs</groupId>
Review Comment:
This is added twice. Why do we need another it since we already have `findbug`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195738159
##########
hadoop-hdds/rocks-native/src/main/patches/rocks-native.patch:
##########
@@ -153,36 +144,83 @@ index eefbaaeee..503ec7aa6 100644
}
return Status::OK();
-@@ -478,21 +481,26 @@ Status SstFileDumper::ReadSequential(bool print_kv, uint64_t read_num,
+ }
+
++void write(int value, FILE* file) {
++ char b[4];
++ b[3] = value & 0x000000ff;
++ b[2] = (value & 0x0000ff00) >> 8;
++ b[1] = (value & 0x00ff0000) >> 16;
++ b[0] = (value & 0xff000000) >> 24;
++ std::fwrite(b, 4, 1, file);
++}
++
++void write(const char* value, int length, FILE* file) {
++ write(length, file);
++ fwrite(value, length, 1, file);
++}
++
++void write(const std::string& value, FILE* file) {
++ write(value.data(), (int)value.length(), file);
++
++}
++
++void write(Slice &slice, FILE* file) {
++ int size = (int)slice.size();
++ write(slice.data(), size, file);
++}
++
++void write(SequenceNumber sequenceNumber, FILE* file) {
++
++ char b[8];
++ int idx = 7;
++ while (idx >= 0) {
++ b[idx] = sequenceNumber % 256;
++ sequenceNumber/=256;
Review Comment:
```suggestion
+ sequenceNumber /= 256;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195496271
##########
hadoop-hdds/rocks-native/pom.xml:
##########
@@ -33,6 +33,25 @@
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-io</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.github.spotbugs</groupId>
+ <artifactId>spotbugs-annotations</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-api</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-params</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-all</artifactId>
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195830366
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -180,33 +182,33 @@ public boolean hasNext() {
public T next() {
checkSanityOfProcess();
currentKey = nextKey;
- nextKey = null;
- boolean keyFound = false;
- while (!keyFound) {
- try {
- Optional<Integer> keyLength = getNextNumberInStream();
- if (!keyLength.isPresent()) {
- return getTransformedValue(currentKey);
- }
- String keyStr = readNextNumberOfCharsFromStream(keyLength.get());
- Matcher matcher = PATTERN_MATCHER.matcher(keyStr);
- if (keyStr.length() == keyLength.get() && matcher.find()) {
- Optional<Integer> valueLength = getNextNumberInStream();
- if (valueLength.isPresent()) {
- String valueStr = readNextNumberOfCharsFromStream(
- valueLength.get());
- if (valueStr.length() == valueLength.get()) {
- keyFound = true;
- nextKey = new KeyValue(matcher.group(PATTERN_KEY_GROUP_NUMBER),
- matcher.group(PATTERN_SEQ_GROUP_NUMBER),
- matcher.group(PATTERN_TYPE_GROUP_NUMBER),
- valueStr);
- }
- }
- }
- } catch (IOException e) {
- throw new RuntimeIOException(e);
+ nextKey = Optional.empty();
+ try {
+ Optional<byte[]> key = getNextByteArray();
+ if (!key.isPresent()) {
+ return getTransformedValue(currentKey);
}
+ UnsignedLong sequenceNumber = getNextUnsignedLong()
+ .orElseThrow(() -> new IllegalStateException(
+ String.format("Error while trying to read sequence number" +
+ " for key %s", StringUtils.bytes2String(key.get()))));
+
+ Integer type = getNextNumberInStream()
+ .orElseThrow(() -> new IllegalStateException(
+ String.format("Error while trying to read sequence number for " +
+ "key %s with sequence number %s",
+ StringUtils.bytes2String(key.get()),
+ sequenceNumber.toString())));
+ byte[] val = getNextByteArray().orElseThrow(() ->
+ new IllegalStateException(
+ String.format("Error while trying to read sequence number for " +
+ "key %s with sequence number %s of type %d",
+ StringUtils.bytes2String(key.get()),
+ sequenceNumber.toString(), type)));
+ nextKey = Optional.of(new KeyValue(key.get(), sequenceNumber, type, val));
+ } catch (IOException e) {
+ // TODO [Snapshot] : Throw custom snapshot exception
Review Comment:
done
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -86,51 +81,58 @@ public ManagedSSTDumpIterator(ManagedSSTDumpTool sstDumpTool,
* @return Optional of the integer empty if no integer exists
*/
private Optional<Integer> getNextNumberInStream() throws IOException {
- StringBuilder value = new StringBuilder();
- int val;
- while ((val = processOutput.read()) != -1) {
- if (val >= '0' && val <= '9') {
- value.append((char) val);
- } else if (value.length() > 0) {
- break;
- }
+ int n = processOutput.read(intBuffer, 0, 4);
+ if (n == 4) {
+ return Optional.of(ByteBuffer.wrap(intBuffer).getInt());
+ } else if (n >= 0) {
+ throw new IllegalStateException(String.format("Integer expects " +
+ "4 bytes to be read from the stream, but read only %d bytes", n));
}
- return value.length() > 0 ? Optional.of(Integer.valueOf(value.toString()))
- : Optional.empty();
+ return Optional.empty();
}
- /**
- * Reads the next n chars from the stream & makes a string.
- *
- * @param numberOfChars
- * @return String of next chars read
- * @throws IOException
- */
- private String readNextNumberOfCharsFromStream(int numberOfChars)
- throws IOException {
- StringBuilder value = new StringBuilder();
- while (numberOfChars > 0) {
- int noOfCharsRead = processOutput.read(charBuffer, 0,
- Math.min(numberOfChars, charBuffer.length));
- if (noOfCharsRead == -1) {
- break;
+ private Optional<byte[]> getNextByteArray() throws IOException {
+ Optional<Integer> size = getNextNumberInStream();
+ if (size.isPresent()) {
+ byte[] b = new byte[size.get()];
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191754236
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -86,51 +79,47 @@ public ManagedSSTDumpIterator(ManagedSSTDumpTool sstDumpTool,
* @return Optional of the integer empty if no integer exists
*/
private Optional<Integer> getNextNumberInStream() throws IOException {
- StringBuilder value = new StringBuilder();
- int val;
- while ((val = processOutput.read()) != -1) {
- if (val >= '0' && val <= '9') {
- value.append((char) val);
- } else if (value.length() > 0) {
- break;
- }
+ int n = processOutput.read(intBuffer, 0, 4);
+ if (n == 4) {
Review Comment:
yeah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191694390
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/PipeInputStream.java:
##########
@@ -65,11 +65,13 @@ public int read() {
this.close();
return -1;
}
- if (numberOfBytesLeftToRead == 0) {
+ while (numberOfBytesLeftToRead == 0) {
numberOfBytesLeftToRead = readInternal(byteBuffer, capacity,
nativeHandle);
index = 0;
- return read();
+ if (numberOfBytesLeftToRead != 0) {
Review Comment:
yeah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1192611717
##########
hadoop-hdds/rocks-native/pom.xml:
##########
@@ -33,6 +33,25 @@
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-io</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.github.spotbugs</groupId>
+ <artifactId>spotbugs-annotations</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-api</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-params</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-all</artifactId>
Review Comment:
I don't think you need this one. You can just use `mockito-junit-jupiter`.
Also limit the scope to `test`.
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
Review Comment:
No need to close it explicitly since you are using `try-with-resources`.
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedOptions options = new ManagedOptions();
+ ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue> iterator =
+ new ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue>(tool,
+ file.getAbsolutePath(), options) {
+
+ @Override
+ protected KeyValue getTransformedValue(Optional<KeyValue> value) {
+ return value.orElse(null);
+ }
+ }) {
+ while (iterator.hasNext()) {
+ ManagedSSTDumpIterator.KeyValue r = iterator.next();
+ Pair<String, Integer> recordKey = Pair.of(new String(r.getKey(),
+ StandardCharsets.UTF_8), r.getType());
+ Assertions.assertTrue(keys.containsKey(recordKey));
+ Assertions.assertEquals(
+ Optional.ofNullable(keys.get(recordKey)).orElse(""),
+ new String(r.getValue(), StandardCharsets.UTF_8));
+ keys.remove(recordKey);
+ }
+ Assertions.assertEquals(0, keys.size());
+ }
+ executorService.shutdown();
+ }
+ }
+
+ private static Stream<? extends Arguments> keyValueFormatArgs() {
+ return Stream.of(
+ Arguments.of(
+ Named.of("Key starting with a single quote",
+ "'key%1$d=>"),
+ Named.of("Value starting with a number ending with a" +
+ " single quote", "%1$dvalue'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number", "%1$dvalue%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& containing null character & new line character",
+ "%1$dvalue\n\0%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number & containing" +
+ " a null character", "key\0%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$dvalue\r%1$d")
+ )
+ );
+ }
+
+ private static byte[] getBytes(Integer val) {
Review Comment:
For int and long, you can either one of the below:
* Option 1: Use [IntegerCodec](https://github.com/apache/ozone/blob/75f5a28f376516e7252e750e86a10c2f9d874d5a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/IntegerCodec.java#L29) and [LongCodec](https://github.com/apache/ozone/blob/75f5a28f376516e7252e750e86a10c2f9d874d5a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/LongCodec.java#L29) respectively
* Option 2: Use `ByteBuffer.allocate(4).putInt().array()` and `ByteBuffer.allocate(8).putLong()` respectively.
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -180,33 +182,32 @@ public boolean hasNext() {
public T next() {
checkSanityOfProcess();
currentKey = nextKey;
- nextKey = null;
- boolean keyFound = false;
- while (!keyFound) {
- try {
- Optional<Integer> keyLength = getNextNumberInStream();
- if (!keyLength.isPresent()) {
- return getTransformedValue(currentKey);
- }
- String keyStr = readNextNumberOfCharsFromStream(keyLength.get());
- Matcher matcher = PATTERN_MATCHER.matcher(keyStr);
- if (keyStr.length() == keyLength.get() && matcher.find()) {
- Optional<Integer> valueLength = getNextNumberInStream();
- if (valueLength.isPresent()) {
- String valueStr = readNextNumberOfCharsFromStream(
- valueLength.get());
- if (valueStr.length() == valueLength.get()) {
- keyFound = true;
- nextKey = new KeyValue(matcher.group(PATTERN_KEY_GROUP_NUMBER),
- matcher.group(PATTERN_SEQ_GROUP_NUMBER),
- matcher.group(PATTERN_TYPE_GROUP_NUMBER),
- valueStr);
- }
- }
- }
- } catch (IOException e) {
- throw new RuntimeIOException(e);
+ nextKey = Optional.empty();
+ try {
+ Optional<byte[]> key = getNextByteArray();
+ if (!key.isPresent()) {
+ return getTransformedValue(currentKey);
}
+ UnsignedLong sequenceNumber = getNextUnsignedLong()
Review Comment:
I don't think you want to throw an exception in case `getNextUnsignedLong` returns `empty()`. If that's the case, then `getNextUnsignedLong` should throw exception instead of returning `Optional.empty()`.
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -244,40 +245,46 @@ protected void finalize() throws Throwable {
* Class containing Parsed KeyValue Record from Sst Dumptool output.
*/
public static final class KeyValue {
- private String key;
- private Integer sequence;
- private Integer type;
- private String value;
+ private final byte[] key;
+ private final UnsignedLong sequence;
+ private final Integer type;
+ private final byte[] value;
- private KeyValue(String key, String sequence, String type,
- String value) {
+ private KeyValue(byte[] key, UnsignedLong sequence, Integer type,
+ byte[] value) {
this.key = key;
- this.sequence = Integer.valueOf(sequence);
- this.type = Integer.valueOf(type);
+ this.sequence = sequence;
+ this.type = type;
this.value = value;
}
- public String getKey() {
+ @SuppressFBWarnings("EI_EXPOSE_REP")
+ public byte[] getKey() {
return key;
}
- public Integer getSequence() {
+ public UnsignedLong getSequence() {
return sequence;
}
public Integer getType() {
return type;
}
- public String getValue() {
+ @SuppressFBWarnings("EI_EXPOSE_REP")
+ public byte[] getValue() {
return value;
}
@Override
public String toString() {
- return "KeyValue{" + "key='" + key + '\'' + ", sequence=" + sequence +
- ", type=" + type + ", value='" + value + '\'' + '}';
+ return "KeyValue{" +
+ "key=" + StringUtils.bytes2String(key) +
Review Comment:
Can `key` and `value` be null? If so, `StringUtils.bytes2String()` will throw NPE if I'm not wrong.
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedOptions options = new ManagedOptions();
+ ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue> iterator =
+ new ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue>(tool,
+ file.getAbsolutePath(), options) {
+
+ @Override
+ protected KeyValue getTransformedValue(Optional<KeyValue> value) {
+ return value.orElse(null);
+ }
+ }) {
+ while (iterator.hasNext()) {
+ ManagedSSTDumpIterator.KeyValue r = iterator.next();
+ Pair<String, Integer> recordKey = Pair.of(new String(r.getKey(),
+ StandardCharsets.UTF_8), r.getType());
+ Assertions.assertTrue(keys.containsKey(recordKey));
+ Assertions.assertEquals(
+ Optional.ofNullable(keys.get(recordKey)).orElse(""),
+ new String(r.getValue(), StandardCharsets.UTF_8));
+ keys.remove(recordKey);
+ }
+ Assertions.assertEquals(0, keys.size());
+ }
+ executorService.shutdown();
+ }
+ }
+
+ private static Stream<? extends Arguments> keyValueFormatArgs() {
+ return Stream.of(
+ Arguments.of(
+ Named.of("Key starting with a single quote",
+ "'key%1$d=>"),
+ Named.of("Value starting with a number ending with a" +
+ " single quote", "%1$dvalue'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number", "%1$dvalue%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& containing null character & new line character",
+ "%1$dvalue\n\0%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number & containing" +
+ " a null character", "key\0%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$dvalue\r%1$d")
+ )
+ );
+ }
+
+ private static byte[] getBytes(Integer val) {
+ byte[] b = new byte[4];
+ for (int i = 3; i >= 0; i--) {
+ b[i] = val.byteValue();
+ val = val >> 8;
+ }
+ return b;
+ }
+
+ private static byte[] getBytes(Long val) {
+ byte[] b = new byte[8];
+ for (int i = 7; i >= 0; i--) {
+ b[i] = (byte)(val & 0xff);
+ val = val >> 8;
+ }
+ return b;
+ }
+
+ private static byte[] getBytes(String val) {
+ byte[] b = new byte[val.length()];
+ for (int i = 0; i < val.length(); i++) {
+ b[i] = (byte) val.charAt(i);
+ }
+ return b;
+ }
+
+ private static Stream<? extends Arguments> invalidPipeInputStreamBytes() {
+ return Stream.of(
+ Arguments.of(Named.of("Invalid 3 byte integer",
+ new byte[] {0, 0, 0})),
+ Arguments.of(Named.of("Invalid 2 byte integer",
+ new byte[] {0, 0})),
+ Arguments.of(Named.of("Invalid 1 byte integer",
+ new byte[] {0, 0})),
+ Arguments.of(Named.of("Invalid key name length",
+ Bytes.concat(getBytes(4), getBytes("key")))),
+ Arguments.of(Named.of("Invalid Unsigned Long length",
+ Bytes.concat(getBytes(4), getBytes("key1"),
+ new byte[]{0, 0}))),
+ Arguments.of(Named.of("Invalid Sequence number",
+ Bytes.concat(getBytes(4), getBytes("key1")))),
+ Arguments.of(Named.of("Invalid Type",
+ Bytes.concat(getBytes(4), getBytes("key1"),
+ getBytes(4L)))),
+ Arguments.of(Named.of("Invalid Value",
+ Bytes.concat(getBytes(4), getBytes("key"),
+ getBytes(4L), getBytes(0)))),
+ Arguments.of(Named.of("Invalid Value length",
+ Bytes.concat(getBytes(4), getBytes("key"),
+ getBytes(4L), getBytes(1), getBytes(6),
+ getBytes("val"))))
+ );
+ }
+
+
Review Comment:
nit: Remove this extra lines.
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -180,33 +182,32 @@ public boolean hasNext() {
public T next() {
checkSanityOfProcess();
currentKey = nextKey;
- nextKey = null;
- boolean keyFound = false;
- while (!keyFound) {
- try {
- Optional<Integer> keyLength = getNextNumberInStream();
- if (!keyLength.isPresent()) {
- return getTransformedValue(currentKey);
- }
- String keyStr = readNextNumberOfCharsFromStream(keyLength.get());
- Matcher matcher = PATTERN_MATCHER.matcher(keyStr);
- if (keyStr.length() == keyLength.get() && matcher.find()) {
- Optional<Integer> valueLength = getNextNumberInStream();
- if (valueLength.isPresent()) {
- String valueStr = readNextNumberOfCharsFromStream(
- valueLength.get());
- if (valueStr.length() == valueLength.get()) {
- keyFound = true;
- nextKey = new KeyValue(matcher.group(PATTERN_KEY_GROUP_NUMBER),
- matcher.group(PATTERN_SEQ_GROUP_NUMBER),
- matcher.group(PATTERN_TYPE_GROUP_NUMBER),
- valueStr);
- }
- }
- }
- } catch (IOException e) {
- throw new RuntimeIOException(e);
+ nextKey = Optional.empty();
+ try {
+ Optional<byte[]> key = getNextByteArray();
+ if (!key.isPresent()) {
+ return getTransformedValue(currentKey);
}
+ UnsignedLong sequenceNumber = getNextUnsignedLong()
+ .orElseThrow(() -> new IllegalStateException(
+ String.format("Error while trying to read sequence number" +
+ " for key %s", StringUtils.bytes2String(key.get()))));
+
+ Integer type = getNextNumberInStream()
+ .orElseThrow(() -> new IllegalStateException(
+ String.format("Error while trying to read sequence number for " +
+ "key %s with sequence number %s",
+ StringUtils.bytes2String(key.get()),
+ sequenceNumber.toString())));
+ byte[] val = getNextByteArray().orElseThrow(() ->
+ new IllegalStateException(
+ String.format("Error while trying to read sequence number for " +
+ "key %s with sequence number %s of type %d",
+ StringUtils.bytes2String(key.get()),
+ sequenceNumber.toString(), type)));
+ nextKey = Optional.of(new KeyValue(key.get(), sequenceNumber, type, val));
+ } catch (IOException e) {
+ throw new RuntimeIOException(e);
Review Comment:
nit: add a TODO (with snapshot tag) to throw custom exception.
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedOptions options = new ManagedOptions();
+ ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue> iterator =
+ new ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue>(tool,
+ file.getAbsolutePath(), options) {
+
+ @Override
+ protected KeyValue getTransformedValue(Optional<KeyValue> value) {
+ return value.orElse(null);
+ }
+ }) {
+ while (iterator.hasNext()) {
+ ManagedSSTDumpIterator.KeyValue r = iterator.next();
+ Pair<String, Integer> recordKey = Pair.of(new String(r.getKey(),
+ StandardCharsets.UTF_8), r.getType());
+ Assertions.assertTrue(keys.containsKey(recordKey));
+ Assertions.assertEquals(
+ Optional.ofNullable(keys.get(recordKey)).orElse(""),
+ new String(r.getValue(), StandardCharsets.UTF_8));
+ keys.remove(recordKey);
+ }
+ Assertions.assertEquals(0, keys.size());
+ }
+ executorService.shutdown();
+ }
+ }
+
+ private static Stream<? extends Arguments> keyValueFormatArgs() {
+ return Stream.of(
+ Arguments.of(
+ Named.of("Key starting with a single quote",
+ "'key%1$d=>"),
+ Named.of("Value starting with a number ending with a" +
+ " single quote", "%1$dvalue'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number", "%1$dvalue%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& containing null character & new line character",
+ "%1$dvalue\n\0%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number & containing" +
+ " a null character", "key\0%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$dvalue\r%1$d")
+ )
+ );
+ }
+
+ private static byte[] getBytes(Integer val) {
+ byte[] b = new byte[4];
+ for (int i = 3; i >= 0; i--) {
+ b[i] = val.byteValue();
+ val = val >> 8;
+ }
+ return b;
+ }
+
+ private static byte[] getBytes(Long val) {
+ byte[] b = new byte[8];
+ for (int i = 7; i >= 0; i--) {
+ b[i] = (byte)(val & 0xff);
+ val = val >> 8;
+ }
+ return b;
+ }
+
+ private static byte[] getBytes(String val) {
Review Comment:
Use `StringUtils.string2Bytes()` instead of creating new one.
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedOptions options = new ManagedOptions();
Review Comment:
Please avoid nested try. Please change it to this way.
```
try {
// First try
try {
// Second try
}
```
Or extract out the code to separate functions.
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSstFileReader.java:
##########
@@ -140,8 +141,9 @@ protected ClosableIterator<String> getKeyIteratorForFile(String file)
return new ManagedSSTDumpIterator<String>(sstDumpTool, file,
options) {
@Override
- protected String getTransformedValue(KeyValue value) {
- return value.getKey();
+ protected String getTransformedValue(Optional<KeyValue> value) {
+ return value.map(v -> new String(v.getKey(), UTF_8))
Review Comment:
nit:
```suggestion
return value.map(v -> StringUtils.bytes2String(v.getKey())).orElse(null);
```
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedOptions options = new ManagedOptions();
+ ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue> iterator =
+ new ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue>(tool,
+ file.getAbsolutePath(), options) {
+
+ @Override
+ protected KeyValue getTransformedValue(Optional<KeyValue> value) {
+ return value.orElse(null);
+ }
+ }) {
+ while (iterator.hasNext()) {
+ ManagedSSTDumpIterator.KeyValue r = iterator.next();
+ Pair<String, Integer> recordKey = Pair.of(new String(r.getKey(),
+ StandardCharsets.UTF_8), r.getType());
+ Assertions.assertTrue(keys.containsKey(recordKey));
+ Assertions.assertEquals(
+ Optional.ofNullable(keys.get(recordKey)).orElse(""),
+ new String(r.getValue(), StandardCharsets.UTF_8));
+ keys.remove(recordKey);
+ }
+ Assertions.assertEquals(0, keys.size());
+ }
+ executorService.shutdown();
+ }
+ }
+
+ private static Stream<? extends Arguments> keyValueFormatArgs() {
+ return Stream.of(
+ Arguments.of(
+ Named.of("Key starting with a single quote",
+ "'key%1$d=>"),
+ Named.of("Value starting with a number ending with a" +
+ " single quote", "%1$dvalue'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number", "%1$dvalue%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& containing null character & new line character",
+ "%1$dvalue\n\0%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number & containing" +
+ " a null character", "key\0%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$dvalue\r%1$d")
+ )
+ );
+ }
+
+ private static byte[] getBytes(Integer val) {
+ byte[] b = new byte[4];
+ for (int i = 3; i >= 0; i--) {
+ b[i] = val.byteValue();
+ val = val >> 8;
+ }
+ return b;
+ }
+
+ private static byte[] getBytes(Long val) {
+ byte[] b = new byte[8];
+ for (int i = 7; i >= 0; i--) {
+ b[i] = (byte)(val & 0xff);
+ val = val >> 8;
+ }
+ return b;
+ }
+
+ private static byte[] getBytes(String val) {
+ byte[] b = new byte[val.length()];
+ for (int i = 0; i < val.length(); i++) {
+ b[i] = (byte) val.charAt(i);
+ }
+ return b;
+ }
+
+ private static Stream<? extends Arguments> invalidPipeInputStreamBytes() {
+ return Stream.of(
+ Arguments.of(Named.of("Invalid 3 byte integer",
+ new byte[] {0, 0, 0})),
+ Arguments.of(Named.of("Invalid 2 byte integer",
+ new byte[] {0, 0})),
+ Arguments.of(Named.of("Invalid 1 byte integer",
+ new byte[] {0, 0})),
+ Arguments.of(Named.of("Invalid key name length",
+ Bytes.concat(getBytes(4), getBytes("key")))),
+ Arguments.of(Named.of("Invalid Unsigned Long length",
+ Bytes.concat(getBytes(4), getBytes("key1"),
+ new byte[]{0, 0}))),
+ Arguments.of(Named.of("Invalid Sequence number",
+ Bytes.concat(getBytes(4), getBytes("key1")))),
+ Arguments.of(Named.of("Invalid Type",
+ Bytes.concat(getBytes(4), getBytes("key1"),
+ getBytes(4L)))),
+ Arguments.of(Named.of("Invalid Value",
+ Bytes.concat(getBytes(4), getBytes("key"),
+ getBytes(4L), getBytes(0)))),
+ Arguments.of(Named.of("Invalid Value length",
+ Bytes.concat(getBytes(4), getBytes("key"),
+ getBytes(4L), getBytes(1), getBytes(6),
+ getBytes("val"))))
+ );
+ }
+
+
+
+ @Native("Managed Rocks Tools")
+ @ParameterizedTest
+ @MethodSource("keyValueFormatArgs")
+ public void testSSTDumpIteratorWithKeyFormat(String keyFormat,
+ String valueFormat) throws Exception {
+ TreeMap<Pair<String, Integer>, String> keys =
+ IntStream.range(0, 100).boxed().collect(
+ Collectors.toMap(
+ i -> Pair.of(String.format(keyFormat, i), i % 2),
+ i -> i % 2 == 0 ? "" : String.format(valueFormat, i),
+ (v1, v2) -> v2,
+ TreeMap::new));
+ testSSTDumpIteratorWithKeys(keys);
Review Comment:
It is not a good idea to delegate one test:`testSSTDumpIteratorWithKeyFormat` on another: `testSSTDumpIteratorWithKeys`.
If you want, can divide the big test into parts like:
```
public void test() {
- Given
// Test setup and mocking. Can be another method, if needed.
- When
// method call writing test for. Can be another method, if needed.
- Then
// Assertions and verifications. Can be another method, if needed.
}
```
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -86,51 +79,47 @@ public ManagedSSTDumpIterator(ManagedSSTDumpTool sstDumpTool,
* @return Optional of the integer empty if no integer exists
*/
private Optional<Integer> getNextNumberInStream() throws IOException {
- StringBuilder value = new StringBuilder();
- int val;
- while ((val = processOutput.read()) != -1) {
- if (val >= '0' && val <= '9') {
- value.append((char) val);
- } else if (value.length() > 0) {
- break;
- }
+ int n = processOutput.read(intBuffer, 0, 4);
+ if (n == 4) {
Review Comment:
Thanks
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSstFileReader.java:
##########
@@ -216,7 +215,8 @@ protected abstract ClosableIterator<T> getKeyIteratorForFile(String file)
public boolean hasNext() {
try {
do {
- if (currentFileIterator.hasNext()) {
+ if (!Objects.isNull(currentFileIterator) &&
Review Comment:
nit: no need to use double negation.
```suggestion
if (Objects.nonNull(currentFileIterator) &&
```
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/PipeInputStream.java:
##########
@@ -65,11 +65,13 @@ public int read() {
this.close();
return -1;
}
- if (numberOfBytesLeftToRead == 0) {
+ while (numberOfBytesLeftToRead == 0) {
numberOfBytesLeftToRead = readInternal(byteBuffer, capacity,
nativeHandle);
index = 0;
- return read();
+ if (numberOfBytesLeftToRead != 0) {
Review Comment:
Does it mean it can return 0 or -1? If so, we should handle both cases. If not, then it is OK to leave as it it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1192790378
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -180,33 +182,32 @@ public boolean hasNext() {
public T next() {
checkSanityOfProcess();
currentKey = nextKey;
- nextKey = null;
- boolean keyFound = false;
- while (!keyFound) {
- try {
- Optional<Integer> keyLength = getNextNumberInStream();
- if (!keyLength.isPresent()) {
- return getTransformedValue(currentKey);
- }
- String keyStr = readNextNumberOfCharsFromStream(keyLength.get());
- Matcher matcher = PATTERN_MATCHER.matcher(keyStr);
- if (keyStr.length() == keyLength.get() && matcher.find()) {
- Optional<Integer> valueLength = getNextNumberInStream();
- if (valueLength.isPresent()) {
- String valueStr = readNextNumberOfCharsFromStream(
- valueLength.get());
- if (valueStr.length() == valueLength.get()) {
- keyFound = true;
- nextKey = new KeyValue(matcher.group(PATTERN_KEY_GROUP_NUMBER),
- matcher.group(PATTERN_SEQ_GROUP_NUMBER),
- matcher.group(PATTERN_TYPE_GROUP_NUMBER),
- valueStr);
- }
- }
- }
- } catch (IOException e) {
- throw new RuntimeIOException(e);
+ nextKey = Optional.empty();
+ try {
+ Optional<byte[]> key = getNextByteArray();
+ if (!key.isPresent()) {
+ return getTransformedValue(currentKey);
}
+ UnsignedLong sequenceNumber = getNextUnsignedLong()
Review Comment:
Ah, I see. So if key is present, then `sequenceNumber`, `type`, and `val` should be present otherwise throw an exception. It makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195744685
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -86,51 +81,58 @@ public ManagedSSTDumpIterator(ManagedSSTDumpTool sstDumpTool,
* @return Optional of the integer empty if no integer exists
*/
private Optional<Integer> getNextNumberInStream() throws IOException {
- StringBuilder value = new StringBuilder();
- int val;
- while ((val = processOutput.read()) != -1) {
- if (val >= '0' && val <= '9') {
- value.append((char) val);
- } else if (value.length() > 0) {
- break;
- }
+ int n = processOutput.read(intBuffer, 0, 4);
+ if (n == 4) {
+ return Optional.of(ByteBuffer.wrap(intBuffer).getInt());
+ } else if (n >= 0) {
+ throw new IllegalStateException(String.format("Integer expects " +
+ "4 bytes to be read from the stream, but read only %d bytes", n));
}
- return value.length() > 0 ? Optional.of(Integer.valueOf(value.toString()))
- : Optional.empty();
+ return Optional.empty();
}
- /**
- * Reads the next n chars from the stream & makes a string.
- *
- * @param numberOfChars
- * @return String of next chars read
- * @throws IOException
- */
- private String readNextNumberOfCharsFromStream(int numberOfChars)
- throws IOException {
- StringBuilder value = new StringBuilder();
- while (numberOfChars > 0) {
- int noOfCharsRead = processOutput.read(charBuffer, 0,
- Math.min(numberOfChars, charBuffer.length));
- if (noOfCharsRead == -1) {
- break;
+ private Optional<byte[]> getNextByteArray() throws IOException {
+ Optional<Integer> size = getNextNumberInStream();
+ if (size.isPresent()) {
+ byte[] b = new byte[size.get()];
Review Comment:
```suggestion
if (size.isPresent()) {
if (LOG.isDebugEnabled()) {
LOG.debug("Size of byte array: {}", size.get());
}
byte[] b = new byte[size.get()];
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195726989
##########
hadoop-hdds/rocks-native/pom.xml:
##########
@@ -345,6 +365,14 @@
</includes>
</configuration>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration>
+ <groups>native</groups>
+ <argLine>${maven-surefire-plugin.argLine} @{argLine} -Djava.library.path=${project.build.directory}/native/rocksdb</argLine>
Review Comment:
nit: extra space?
```suggestion
<argLine>${maven-surefire-plugin.argLine} @{argLine} -Djava.library.path=${project.build.directory}/native/rocksdb</argLine>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195830511
##########
hadoop-hdds/rocks-native/src/main/patches/rocks-native.patch:
##########
@@ -153,36 +144,83 @@ index eefbaaeee..503ec7aa6 100644
}
return Status::OK();
-@@ -478,21 +481,26 @@ Status SstFileDumper::ReadSequential(bool print_kv, uint64_t read_num,
+ }
+
++void write(int value, FILE* file) {
++ char b[4];
++ b[3] = value & 0x000000ff;
++ b[2] = (value & 0x0000ff00) >> 8;
++ b[1] = (value & 0x00ff0000) >> 16;
++ b[0] = (value & 0xff000000) >> 24;
++ std::fwrite(b, 4, 1, file);
++}
++
++void write(const char* value, int length, FILE* file) {
++ write(length, file);
++ fwrite(value, length, 1, file);
++}
++
++void write(const std::string& value, FILE* file) {
++ write(value.data(), (int)value.length(), file);
++
++}
++
++void write(Slice &slice, FILE* file) {
++ int size = (int)slice.size();
++ write(slice.data(), size, file);
++}
++
++void write(SequenceNumber sequenceNumber, FILE* file) {
++
++ char b[8];
++ int idx = 7;
++ while (idx >= 0) {
++ b[idx] = sequenceNumber % 256;
++ sequenceNumber/=256;
Review Comment:
done
##########
hadoop-hdds/rocks-native/src/main/patches/rocks-native.patch:
##########
@@ -153,36 +144,83 @@ index eefbaaeee..503ec7aa6 100644
}
return Status::OK();
-@@ -478,21 +481,26 @@ Status SstFileDumper::ReadSequential(bool print_kv, uint64_t read_num,
+ }
+
++void write(int value, FILE* file) {
++ char b[4];
++ b[3] = value & 0x000000ff;
++ b[2] = (value & 0x0000ff00) >> 8;
++ b[1] = (value & 0x00ff0000) >> 16;
++ b[0] = (value & 0xff000000) >> 24;
++ std::fwrite(b, 4, 1, file);
++}
++
++void write(const char* value, int length, FILE* file) {
++ write(length, file);
++ fwrite(value, length, 1, file);
++}
++
++void write(const std::string& value, FILE* file) {
++ write(value.data(), (int)value.length(), file);
++
++}
++
++void write(Slice &slice, FILE* file) {
++ int size = (int)slice.size();
++ write(slice.data(), size, file);
++}
++
++void write(SequenceNumber sequenceNumber, FILE* file) {
++
++ char b[8];
++ int idx = 7;
++ while (idx >= 0) {
++ b[idx] = sequenceNumber % 256;
++ sequenceNumber/=256;
++ idx -= 1;
++ }
++ fwrite(b, 8, 1, file);
++}
++
++void write(ParsedInternalKey &key, FILE* file) {
++ write(key.user_key, file);
++ write(key.sequence, file);
++ write(static_cast<int>(key.type), file);
++}
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195450535
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSstFileReader.java:
##########
@@ -216,7 +215,8 @@ protected abstract ClosableIterator<T> getKeyIteratorForFile(String file)
public boolean hasNext() {
try {
do {
- if (currentFileIterator.hasNext()) {
+ if (!Objects.isNull(currentFileIterator) &&
Review Comment:
done
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSstFileReader.java:
##########
@@ -140,8 +141,9 @@ protected ClosableIterator<String> getKeyIteratorForFile(String file)
return new ManagedSSTDumpIterator<String>(sstDumpTool, file,
options) {
@Override
- protected String getTransformedValue(KeyValue value) {
- return value.getKey();
+ protected String getTransformedValue(Optional<KeyValue> value) {
+ return value.map(v -> new String(v.getKey(), UTF_8))
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191133403
##########
hadoop-hdds/rocks-native/pom.xml:
##########
@@ -436,5 +449,24 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>noNativeTest</id>
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191569706
##########
hadoop-hdds/rocks-native/pom.xml:
##########
@@ -33,6 +33,31 @@
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-io</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.github.spotbugs</groupId>
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4584:
URL: https://github.com/apache/ozone/pull/4584#issuecomment-1553520314
@prashantpogde please remove unnecessary details from the commit message when merging PRs. "Address review comments", "Fix checkstyle" and similar items from individual commit messages do not add any value, but make git log for everyone unnecessarily long.
```
$ git show beec17828524cb53ccf496e23f9e0cb93b3cf7a0
commit beec17828524cb53ccf496e23f9e0cb93b3cf7a0 (HEAD -> master, origin/master, origin/HEAD)
Author: Swaminathan Balachandran <47...@users.noreply.github.com>
Date: Thu May 18 11:14:57 2023 -0700
HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool (#4584)
* HDDS-8306. [Snapshot] Add tests for SST dump tool
* HDDS-8306. Address review comments
* HDDS-8306. Fix Failures
* HDDS-8306. Fix checkstyle
* HDDS-8306. Address review comments
* HDDS-8306. Address review comments
* HDDS-8306. Address review comments
* HDDS-8306. Address review comments
* HDDS-8306. Address review comments
* HDDS-8306. Address review comments
* HDDS-8306. Fix checkstyle
* HDDS-8306. Address review comments
* HDDS-8306. Address review comments
diff --git hadoop-hdds/rocks-native/pom.xml hadoop-hdds/rocks-native/pom.xml
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195450829
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedOptions options = new ManagedOptions();
+ ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue> iterator =
+ new ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue>(tool,
+ file.getAbsolutePath(), options) {
+
+ @Override
+ protected KeyValue getTransformedValue(Optional<KeyValue> value) {
+ return value.orElse(null);
+ }
+ }) {
+ while (iterator.hasNext()) {
+ ManagedSSTDumpIterator.KeyValue r = iterator.next();
+ Pair<String, Integer> recordKey = Pair.of(new String(r.getKey(),
+ StandardCharsets.UTF_8), r.getType());
+ Assertions.assertTrue(keys.containsKey(recordKey));
+ Assertions.assertEquals(
+ Optional.ofNullable(keys.get(recordKey)).orElse(""),
+ new String(r.getValue(), StandardCharsets.UTF_8));
+ keys.remove(recordKey);
+ }
+ Assertions.assertEquals(0, keys.size());
+ }
+ executorService.shutdown();
+ }
+ }
+
+ private static Stream<? extends Arguments> keyValueFormatArgs() {
+ return Stream.of(
+ Arguments.of(
+ Named.of("Key starting with a single quote",
+ "'key%1$d=>"),
+ Named.of("Value starting with a number ending with a" +
+ " single quote", "%1$dvalue'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number", "%1$dvalue%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& containing null character & new line character",
+ "%1$dvalue\n\0%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number & containing" +
+ " a null character", "key\0%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$dvalue\r%1$d")
+ )
+ );
+ }
+
+ private static byte[] getBytes(Integer val) {
+ byte[] b = new byte[4];
+ for (int i = 3; i >= 0; i--) {
+ b[i] = val.byteValue();
+ val = val >> 8;
+ }
+ return b;
+ }
+
+ private static byte[] getBytes(Long val) {
+ byte[] b = new byte[8];
+ for (int i = 7; i >= 0; i--) {
+ b[i] = (byte)(val & 0xff);
+ val = val >> 8;
+ }
+ return b;
+ }
+
+ private static byte[] getBytes(String val) {
+ byte[] b = new byte[val.length()];
+ for (int i = 0; i < val.length(); i++) {
+ b[i] = (byte) val.charAt(i);
+ }
+ return b;
+ }
+
+ private static Stream<? extends Arguments> invalidPipeInputStreamBytes() {
+ return Stream.of(
+ Arguments.of(Named.of("Invalid 3 byte integer",
+ new byte[] {0, 0, 0})),
+ Arguments.of(Named.of("Invalid 2 byte integer",
+ new byte[] {0, 0})),
+ Arguments.of(Named.of("Invalid 1 byte integer",
+ new byte[] {0, 0})),
+ Arguments.of(Named.of("Invalid key name length",
+ Bytes.concat(getBytes(4), getBytes("key")))),
+ Arguments.of(Named.of("Invalid Unsigned Long length",
+ Bytes.concat(getBytes(4), getBytes("key1"),
+ new byte[]{0, 0}))),
+ Arguments.of(Named.of("Invalid Sequence number",
+ Bytes.concat(getBytes(4), getBytes("key1")))),
+ Arguments.of(Named.of("Invalid Type",
+ Bytes.concat(getBytes(4), getBytes("key1"),
+ getBytes(4L)))),
+ Arguments.of(Named.of("Invalid Value",
+ Bytes.concat(getBytes(4), getBytes("key"),
+ getBytes(4L), getBytes(0)))),
+ Arguments.of(Named.of("Invalid Value length",
+ Bytes.concat(getBytes(4), getBytes("key"),
+ getBytes(4L), getBytes(1), getBytes(6),
+ getBytes("val"))))
+ );
+ }
+
+
+
+ @Native("Managed Rocks Tools")
+ @ParameterizedTest
+ @MethodSource("keyValueFormatArgs")
+ public void testSSTDumpIteratorWithKeyFormat(String keyFormat,
+ String valueFormat) throws Exception {
+ TreeMap<Pair<String, Integer>, String> keys =
+ IntStream.range(0, 100).boxed().collect(
+ Collectors.toMap(
+ i -> Pair.of(String.format(keyFormat, i), i % 2),
+ i -> i % 2 == 0 ? "" : String.format(valueFormat, i),
+ (v1, v2) -> v2,
+ TreeMap::new));
+ testSSTDumpIteratorWithKeys(keys);
Review Comment:
done
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedOptions options = new ManagedOptions();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195512495
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -180,33 +182,32 @@ public boolean hasNext() {
public T next() {
checkSanityOfProcess();
currentKey = nextKey;
- nextKey = null;
- boolean keyFound = false;
- while (!keyFound) {
- try {
- Optional<Integer> keyLength = getNextNumberInStream();
- if (!keyLength.isPresent()) {
- return getTransformedValue(currentKey);
- }
- String keyStr = readNextNumberOfCharsFromStream(keyLength.get());
- Matcher matcher = PATTERN_MATCHER.matcher(keyStr);
- if (keyStr.length() == keyLength.get() && matcher.find()) {
- Optional<Integer> valueLength = getNextNumberInStream();
- if (valueLength.isPresent()) {
- String valueStr = readNextNumberOfCharsFromStream(
- valueLength.get());
- if (valueStr.length() == valueLength.get()) {
- keyFound = true;
- nextKey = new KeyValue(matcher.group(PATTERN_KEY_GROUP_NUMBER),
- matcher.group(PATTERN_SEQ_GROUP_NUMBER),
- matcher.group(PATTERN_TYPE_GROUP_NUMBER),
- valueStr);
- }
- }
- }
- } catch (IOException e) {
- throw new RuntimeIOException(e);
+ nextKey = Optional.empty();
+ try {
+ Optional<byte[]> key = getNextByteArray();
+ if (!key.isPresent()) {
+ return getTransformedValue(currentKey);
}
+ UnsignedLong sequenceNumber = getNextUnsignedLong()
+ .orElseThrow(() -> new IllegalStateException(
+ String.format("Error while trying to read sequence number" +
+ " for key %s", StringUtils.bytes2String(key.get()))));
+
+ Integer type = getNextNumberInStream()
+ .orElseThrow(() -> new IllegalStateException(
+ String.format("Error while trying to read sequence number for " +
+ "key %s with sequence number %s",
+ StringUtils.bytes2String(key.get()),
+ sequenceNumber.toString())));
+ byte[] val = getNextByteArray().orElseThrow(() ->
+ new IllegalStateException(
+ String.format("Error while trying to read sequence number for " +
+ "key %s with sequence number %s of type %d",
+ StringUtils.bytes2String(key.get()),
+ sequenceNumber.toString(), type)));
+ nextKey = Optional.of(new KeyValue(key.get(), sequenceNumber, type, val));
+ } catch (IOException e) {
+ throw new RuntimeIOException(e);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195733634
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/NativeLibraryLoader.java:
##########
@@ -132,15 +134,17 @@ private Optional<File> copyResourceFromJarToTemp(final String libraryName)
}
// create a temporary file to copy the library to
- final File temp = File.createTempFile(libraryName, getLibOsSuffix());
+ final File temp = File.createTempFile(libraryName, getLibOsSuffix(),
+ new File(""));
if (!temp.exists()) {
return Optional.empty();
} else {
temp.deleteOnExit();
}
Files.copy(is, temp.toPath(), StandardCopyOption.REPLACE_EXISTING);
- return Optional.ofNullable(temp);
+ ShutdownHookManager.get().addShutdownHook(temp::delete, 1);
Review Comment:
At least define a class local constant like [others does](https://github.com/apache/ozone/blob/297c3c1eb3ffa596d187cfd1618ab82ba3cf5d4d/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L377).
```suggestion
ShutdownHookManager.get().addShutdownHook(temp::delete, LIBRARY_SHUTDOWN_HOOK_PRIORITY);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195733634
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/NativeLibraryLoader.java:
##########
@@ -132,15 +134,17 @@ private Optional<File> copyResourceFromJarToTemp(final String libraryName)
}
// create a temporary file to copy the library to
- final File temp = File.createTempFile(libraryName, getLibOsSuffix());
+ final File temp = File.createTempFile(libraryName, getLibOsSuffix(),
+ new File(""));
if (!temp.exists()) {
return Optional.empty();
} else {
temp.deleteOnExit();
}
Files.copy(is, temp.toPath(), StandardCopyOption.REPLACE_EXISTING);
- return Optional.ofNullable(temp);
+ ShutdownHookManager.get().addShutdownHook(temp::delete, 1);
Review Comment:
At least define a constant like [others does](https://github.com/apache/ozone/blob/297c3c1eb3ffa596d187cfd1618ab82ba3cf5d4d/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L377).
```suggestion
ShutdownHookManager.get().addShutdownHook(temp::delete, LIBRARY_SHUTDOWN_HOOK_PRIORITY);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195737109
##########
hadoop-hdds/rocks-native/src/main/patches/rocks-native.patch:
##########
@@ -153,36 +144,83 @@ index eefbaaeee..503ec7aa6 100644
}
return Status::OK();
-@@ -478,21 +481,26 @@ Status SstFileDumper::ReadSequential(bool print_kv, uint64_t read_num,
+ }
+
++void write(int value, FILE* file) {
++ char b[4];
++ b[3] = value & 0x000000ff;
++ b[2] = (value & 0x0000ff00) >> 8;
++ b[1] = (value & 0x00ff0000) >> 16;
++ b[0] = (value & 0xff000000) >> 24;
++ std::fwrite(b, 4, 1, file);
++}
++
++void write(const char* value, int length, FILE* file) {
++ write(length, file);
++ fwrite(value, length, 1, file);
++}
++
++void write(const std::string& value, FILE* file) {
++ write(value.data(), (int)value.length(), file);
++
++}
Review Comment:
nit
```suggestion
+void write(const std::string& value, FILE* file) {
+ write(value.data(), (int)value.length(), file);
+}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195451674
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedOptions options = new ManagedOptions();
+ ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue> iterator =
+ new ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue>(tool,
+ file.getAbsolutePath(), options) {
+
+ @Override
+ protected KeyValue getTransformedValue(Optional<KeyValue> value) {
+ return value.orElse(null);
+ }
+ }) {
+ while (iterator.hasNext()) {
+ ManagedSSTDumpIterator.KeyValue r = iterator.next();
+ Pair<String, Integer> recordKey = Pair.of(new String(r.getKey(),
+ StandardCharsets.UTF_8), r.getType());
+ Assertions.assertTrue(keys.containsKey(recordKey));
+ Assertions.assertEquals(
+ Optional.ofNullable(keys.get(recordKey)).orElse(""),
+ new String(r.getValue(), StandardCharsets.UTF_8));
+ keys.remove(recordKey);
+ }
+ Assertions.assertEquals(0, keys.size());
+ }
+ executorService.shutdown();
+ }
+ }
+
+ private static Stream<? extends Arguments> keyValueFormatArgs() {
+ return Stream.of(
+ Arguments.of(
+ Named.of("Key starting with a single quote",
+ "'key%1$d=>"),
+ Named.of("Value starting with a number ending with a" +
+ " single quote", "%1$dvalue'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number", "%1$dvalue%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& containing null character & new line character",
+ "%1$dvalue\n\0%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number & containing" +
+ " a null character", "key\0%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$dvalue\r%1$d")
+ )
+ );
+ }
+
+ private static byte[] getBytes(Integer val) {
+ byte[] b = new byte[4];
+ for (int i = 3; i >= 0; i--) {
+ b[i] = val.byteValue();
+ val = val >> 8;
+ }
+ return b;
+ }
+
+ private static byte[] getBytes(Long val) {
+ byte[] b = new byte[8];
+ for (int i = 7; i >= 0; i--) {
+ b[i] = (byte)(val & 0xff);
+ val = val >> 8;
+ }
+ return b;
+ }
+
+ private static byte[] getBytes(String val) {
+ byte[] b = new byte[val.length()];
+ for (int i = 0; i < val.length(); i++) {
+ b[i] = (byte) val.charAt(i);
+ }
+ return b;
+ }
+
+ private static Stream<? extends Arguments> invalidPipeInputStreamBytes() {
+ return Stream.of(
+ Arguments.of(Named.of("Invalid 3 byte integer",
+ new byte[] {0, 0, 0})),
+ Arguments.of(Named.of("Invalid 2 byte integer",
+ new byte[] {0, 0})),
+ Arguments.of(Named.of("Invalid 1 byte integer",
+ new byte[] {0, 0})),
+ Arguments.of(Named.of("Invalid key name length",
+ Bytes.concat(getBytes(4), getBytes("key")))),
+ Arguments.of(Named.of("Invalid Unsigned Long length",
+ Bytes.concat(getBytes(4), getBytes("key1"),
+ new byte[]{0, 0}))),
+ Arguments.of(Named.of("Invalid Sequence number",
+ Bytes.concat(getBytes(4), getBytes("key1")))),
+ Arguments.of(Named.of("Invalid Type",
+ Bytes.concat(getBytes(4), getBytes("key1"),
+ getBytes(4L)))),
+ Arguments.of(Named.of("Invalid Value",
+ Bytes.concat(getBytes(4), getBytes("key"),
+ getBytes(4L), getBytes(0)))),
+ Arguments.of(Named.of("Invalid Value length",
+ Bytes.concat(getBytes(4), getBytes("key"),
+ getBytes(4L), getBytes(1), getBytes(6),
+ getBytes("val"))))
+ );
+ }
+
+
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195455035
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedOptions options = new ManagedOptions();
+ ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue> iterator =
+ new ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue>(tool,
+ file.getAbsolutePath(), options) {
+
+ @Override
+ protected KeyValue getTransformedValue(Optional<KeyValue> value) {
+ return value.orElse(null);
+ }
+ }) {
+ while (iterator.hasNext()) {
+ ManagedSSTDumpIterator.KeyValue r = iterator.next();
+ Pair<String, Integer> recordKey = Pair.of(new String(r.getKey(),
+ StandardCharsets.UTF_8), r.getType());
+ Assertions.assertTrue(keys.containsKey(recordKey));
+ Assertions.assertEquals(
+ Optional.ofNullable(keys.get(recordKey)).orElse(""),
+ new String(r.getValue(), StandardCharsets.UTF_8));
+ keys.remove(recordKey);
+ }
+ Assertions.assertEquals(0, keys.size());
+ }
+ executorService.shutdown();
+ }
+ }
+
+ private static Stream<? extends Arguments> keyValueFormatArgs() {
+ return Stream.of(
+ Arguments.of(
+ Named.of("Key starting with a single quote",
+ "'key%1$d=>"),
+ Named.of("Value starting with a number ending with a" +
+ " single quote", "%1$dvalue'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number", "%1$dvalue%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& containing null character & new line character",
+ "%1$dvalue\n\0%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number & containing" +
+ " a null character", "key\0%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$dvalue\r%1$d")
+ )
+ );
+ }
+
+ private static byte[] getBytes(Integer val) {
+ byte[] b = new byte[4];
+ for (int i = 3; i >= 0; i--) {
+ b[i] = val.byteValue();
+ val = val >> 8;
+ }
+ return b;
+ }
+
+ private static byte[] getBytes(Long val) {
+ byte[] b = new byte[8];
+ for (int i = 7; i >= 0; i--) {
+ b[i] = (byte)(val & 0xff);
+ val = val >> 8;
+ }
+ return b;
+ }
+
+ private static byte[] getBytes(String val) {
Review Comment:
no we cannot do that, it will encode into utf-8. I want the raw character byte array, since I am already prepending the bytes of the length in sst dumptool
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195730545
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/NativeLibraryLoader.java:
##########
@@ -132,15 +134,17 @@ private Optional<File> copyResourceFromJarToTemp(final String libraryName)
}
// create a temporary file to copy the library to
- final File temp = File.createTempFile(libraryName, getLibOsSuffix());
+ final File temp = File.createTempFile(libraryName, getLibOsSuffix(),
+ new File(""));
if (!temp.exists()) {
return Optional.empty();
} else {
temp.deleteOnExit();
}
Files.copy(is, temp.toPath(), StandardCopyOption.REPLACE_EXISTING);
- return Optional.ofNullable(temp);
+ ShutdownHookManager.get().addShutdownHook(temp::delete, 1);
+ return Optional.of(temp);
Review Comment:
`temp` could still be `null` if `File.createTempFile()` above doesn't have the permission to write to current dir right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195744685
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -86,51 +81,58 @@ public ManagedSSTDumpIterator(ManagedSSTDumpTool sstDumpTool,
* @return Optional of the integer empty if no integer exists
*/
private Optional<Integer> getNextNumberInStream() throws IOException {
- StringBuilder value = new StringBuilder();
- int val;
- while ((val = processOutput.read()) != -1) {
- if (val >= '0' && val <= '9') {
- value.append((char) val);
- } else if (value.length() > 0) {
- break;
- }
+ int n = processOutput.read(intBuffer, 0, 4);
+ if (n == 4) {
+ return Optional.of(ByteBuffer.wrap(intBuffer).getInt());
+ } else if (n >= 0) {
+ throw new IllegalStateException(String.format("Integer expects " +
+ "4 bytes to be read from the stream, but read only %d bytes", n));
}
- return value.length() > 0 ? Optional.of(Integer.valueOf(value.toString()))
- : Optional.empty();
+ return Optional.empty();
}
- /**
- * Reads the next n chars from the stream & makes a string.
- *
- * @param numberOfChars
- * @return String of next chars read
- * @throws IOException
- */
- private String readNextNumberOfCharsFromStream(int numberOfChars)
- throws IOException {
- StringBuilder value = new StringBuilder();
- while (numberOfChars > 0) {
- int noOfCharsRead = processOutput.read(charBuffer, 0,
- Math.min(numberOfChars, charBuffer.length));
- if (noOfCharsRead == -1) {
- break;
+ private Optional<byte[]> getNextByteArray() throws IOException {
+ Optional<Integer> size = getNextNumberInStream();
+ if (size.isPresent()) {
+ byte[] b = new byte[size.get()];
Review Comment:
debug logging the byte array size
```suggestion
if (size.isPresent()) {
if (LOG.isDebugEnabled()) {
LOG.debug("Size of byte array: {}", size.get());
}
byte[] b = new byte[size.get()];
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195830635
##########
hadoop-hdds/rocks-native/src/main/patches/rocks-native.patch:
##########
@@ -153,36 +144,83 @@ index eefbaaeee..503ec7aa6 100644
}
return Status::OK();
-@@ -478,21 +481,26 @@ Status SstFileDumper::ReadSequential(bool print_kv, uint64_t read_num,
+ }
+
++void write(int value, FILE* file) {
++ char b[4];
++ b[3] = value & 0x000000ff;
++ b[2] = (value & 0x0000ff00) >> 8;
++ b[1] = (value & 0x00ff0000) >> 16;
++ b[0] = (value & 0xff000000) >> 24;
++ std::fwrite(b, 4, 1, file);
++}
++
++void write(const char* value, int length, FILE* file) {
++ write(length, file);
++ fwrite(value, length, 1, file);
++}
++
++void write(const std::string& value, FILE* file) {
++ write(value.data(), (int)value.length(), file);
++
++}
Review Comment:
done
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/NativeLibraryLoader.java:
##########
@@ -132,15 +134,17 @@ private Optional<File> copyResourceFromJarToTemp(final String libraryName)
}
// create a temporary file to copy the library to
- final File temp = File.createTempFile(libraryName, getLibOsSuffix());
+ final File temp = File.createTempFile(libraryName, getLibOsSuffix(),
+ new File(""));
if (!temp.exists()) {
return Optional.empty();
} else {
temp.deleteOnExit();
}
Files.copy(is, temp.toPath(), StandardCopyOption.REPLACE_EXISTING);
- return Optional.ofNullable(temp);
+ ShutdownHookManager.get().addShutdownHook(temp::delete, 1);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195828191
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/NativeLibraryLoader.java:
##########
@@ -132,15 +134,17 @@ private Optional<File> copyResourceFromJarToTemp(final String libraryName)
}
// create a temporary file to copy the library to
- final File temp = File.createTempFile(libraryName, getLibOsSuffix());
+ final File temp = File.createTempFile(libraryName, getLibOsSuffix(),
+ new File(""));
if (!temp.exists()) {
return Optional.empty();
} else {
temp.deleteOnExit();
}
Files.copy(is, temp.toPath(), StandardCopyOption.REPLACE_EXISTING);
- return Optional.ofNullable(temp);
+ ShutdownHookManager.get().addShutdownHook(temp::delete, 1);
+ return Optional.of(temp);
Review Comment:
it will throw an IOException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Add tests for SST dump tool
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1185594025
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,109 @@
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.IntStream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ Map<Pair<String, Integer>, String> records)
+ throws Exception {
+ Map<Pair<String, Integer>, String> keys = records instanceof TreeMap ?
Review Comment:
Just to clarify, I meant parameterized test of Junit-5 not Junit-4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191755188
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -86,51 +79,47 @@ public ManagedSSTDumpIterator(ManagedSSTDumpTool sstDumpTool,
* @return Optional of the integer empty if no integer exists
*/
private Optional<Integer> getNextNumberInStream() throws IOException {
- StringBuilder value = new StringBuilder();
- int val;
- while ((val = processOutput.read()) != -1) {
- if (val >= '0' && val <= '9') {
- value.append((char) val);
- } else if (value.length() > 0) {
- break;
- }
+ int n = processOutput.read(intBuffer, 0, 4);
+ if (n == 4) {
Review Comment:
Ideally it should not be corrupted, since rocksdb checks the sst file before reading it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191707606
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/PipeInputStream.java:
##########
@@ -65,11 +65,13 @@ public int read() {
this.close();
return -1;
}
- if (numberOfBytesLeftToRead == 0) {
+ while (numberOfBytesLeftToRead == 0) {
numberOfBytesLeftToRead = readInternal(byteBuffer, capacity,
nativeHandle);
index = 0;
- return read();
+ if (numberOfBytesLeftToRead != 0) {
Review Comment:
In that case, shouldn't we skip `read()` if `numberOfBytesLeftToRead == -1`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Add tests for SST dump tool
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191530724
##########
hadoop-hdds/rocks-native/pom.xml:
##########
@@ -33,6 +33,31 @@
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-io</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.github.spotbugs</groupId>
Review Comment:
This is added twice, here and line 56-60. Why do we need another it since we already have `findbug`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191591228
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.ArgumentsProvider;
+import org.junit.jupiter.params.provider.ArgumentsSource;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
Review Comment:
nit: I think it would look nicer if done this way. There is not need to implement custom `ArgumentsProvider`.
```suggestion
private static Stream<Arguments> sstDumpToolScenarios() {
return Stream.of(
Arguments.of("'key%1$d=>", "%1$dvalue'"),
Arguments.of("key%1$d", "%1$dvalue%1$d"),
Arguments.of("'key%1$d", "%1$d'value%1$d'"),
Arguments.of("'key%1$d", "%1$d'value%1$d'"),
Arguments.of("key%1$d", "%1$dvalue\n\0%1$d"),
Arguments.of("key\0%1$d", "%1$dvalue\r%1$d")
);
}
private void populateSstFile(File file,
TreeMap<Pair<String, Integer>, String> keys)
throws RocksDBException {
try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
ManagedOptions managedOptions = new ManagedOptions();
ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
envOptions, managedOptions)) {
sstFileWriter.open(file.getAbsolutePath());
for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
if (entry.getKey().getValue() == 0) {
sstFileWriter.delete(entry.getKey().getKey()
.getBytes(StandardCharsets.UTF_8));
} else {
sstFileWriter.put(entry.getKey().getKey()
.getBytes(StandardCharsets.UTF_8),
entry.getValue().getBytes(StandardCharsets.UTF_8));
}
}
sstFileWriter.finish();
}
}
@Native("Managed Rocks Tools")
@ParameterizedTest
@MethodSource("sstDumpToolScenarios")
public void testSSTDumpIteratorWithKeyFormat(String keyFormat,
String valueFormat)
throws IOException, NativeLibraryNotLoadedException, RocksDBException {
TreeMap<Pair<String, Integer>, String> keys = IntStream.range(0, 100)
.boxed()
.collect(
Collectors.toMap(
i -> Pair.of(String.format(keyFormat, i), i % 2),
i -> i % 2 == 0 ? "" : String.format(valueFormat, i),
(v1, v2) -> v2,
TreeMap::new)
);
File file = File.createTempFile("tmp_sst_file", ".sst");
file.deleteOnExit();
populateSstFile(file, keys);
ExecutorService executorService = new ThreadPoolExecutor(1, 1, 0, SECONDS,
new ArrayBlockingQueue<>(1), new CallerRunsPolicy());
ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue> iterator = null;
try {
iterator = new ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue>(
tool,
file.getAbsolutePath(),
new ManagedOptions()
) {
@Override
protected KeyValue getTransformedValue(
Optional<KeyValue> value) {
return value.orElse(null);
}
};
while (iterator.hasNext()) {
ManagedSSTDumpIterator.KeyValue r = iterator.next();
Pair<String, Integer> recordKey = Pair.of(new String(r.getKey(),
StandardCharsets.UTF_8), r.getType());
Assertions.assertTrue(keys.containsKey(recordKey));
Assertions.assertEquals(
Optional.ofNullable(keys.get(recordKey)).orElse(""),
new String(r.getValue(), StandardCharsets.UTF_8));
keys.remove(recordKey);
}
Assertions.assertEquals(0, keys.size());
} finally {
IOUtils.closeQuietly(iterator);
executorService.shutdown();
}
}
```
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -86,51 +79,47 @@ public ManagedSSTDumpIterator(ManagedSSTDumpTool sstDumpTool,
* @return Optional of the integer empty if no integer exists
*/
private Optional<Integer> getNextNumberInStream() throws IOException {
- StringBuilder value = new StringBuilder();
- int val;
- while ((val = processOutput.read()) != -1) {
- if (val >= '0' && val <= '9') {
- value.append((char) val);
- } else if (value.length() > 0) {
- break;
- }
+ int n = processOutput.read(intBuffer, 0, 4);
+ if (n == 4) {
+ return Optional.of(ByteBuffer.wrap(intBuffer).getInt());
}
- return value.length() > 0 ? Optional.of(Integer.valueOf(value.toString()))
- : Optional.empty();
+ return Optional.empty();
}
- /**
- * Reads the next n chars from the stream & makes a string.
- *
- * @param numberOfChars
- * @return String of next chars read
- * @throws IOException
- */
- private String readNextNumberOfCharsFromStream(int numberOfChars)
- throws IOException {
- StringBuilder value = new StringBuilder();
- while (numberOfChars > 0) {
- int noOfCharsRead = processOutput.read(charBuffer, 0,
- Math.min(numberOfChars, charBuffer.length));
- if (noOfCharsRead == -1) {
- break;
- }
- value.append(charBuffer, 0, noOfCharsRead);
- numberOfChars -= noOfCharsRead;
+ private Optional<byte[]> getNextByteArray() throws IOException {
+ Optional<Integer> size = getNextNumberInStream();
+ if (size.isPresent()) {
+ byte[] b = new byte[size.get()];
+ int n = processOutput.read(b);
+ return n != size.get() ? Optional.empty() : Optional.of(b);
}
+ return Optional.empty();
+ }
- return value.toString();
+ private Optional<UnsignedLong> getNextUnsignedLong() {
+ long val = 0;
+ for (int i = 0; i < 8; i++) {
+ val = val << 8;
+ int nextByte = processOutput.read();
+ if (nextByte < 0) {
+ return Optional.empty();
+ }
+ val += nextByte;
+ }
+ return Optional.of(UnsignedLong.fromLongBits(val));
}
private void init(ManagedSSTDumpTool sstDumpTool, File sstFile,
ManagedOptions options)
throws NativeLibraryNotLoadedException {
- String[] args = {"--file=" + sstFile.getAbsolutePath(), "--command=scan"};
+ String[] args = {"--file=" + sstFile.getAbsolutePath(), "--command=scan",
Review Comment:
```suggestion
String[] args = {"--file=" + sstFile.getAbsolutePath(), "--command=scan",
"--silent"};
```
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -180,33 +169,25 @@ public boolean hasNext() {
public T next() {
checkSanityOfProcess();
currentKey = nextKey;
- nextKey = null;
- boolean keyFound = false;
- while (!keyFound) {
- try {
- Optional<Integer> keyLength = getNextNumberInStream();
- if (!keyLength.isPresent()) {
- return getTransformedValue(currentKey);
- }
- String keyStr = readNextNumberOfCharsFromStream(keyLength.get());
- Matcher matcher = PATTERN_MATCHER.matcher(keyStr);
- if (keyStr.length() == keyLength.get() && matcher.find()) {
- Optional<Integer> valueLength = getNextNumberInStream();
- if (valueLength.isPresent()) {
- String valueStr = readNextNumberOfCharsFromStream(
- valueLength.get());
- if (valueStr.length() == valueLength.get()) {
- keyFound = true;
- nextKey = new KeyValue(matcher.group(PATTERN_KEY_GROUP_NUMBER),
- matcher.group(PATTERN_SEQ_GROUP_NUMBER),
- matcher.group(PATTERN_TYPE_GROUP_NUMBER),
- valueStr);
+ nextKey = Optional.empty();
+ try {
Review Comment:
nit: next() has too much nesting and could be simplified to this.
```suggestion
try {
Optional<byte[]> key = getNextByteArray();
if (!key.isPresent()) {
return getTransformedValue(currentKey);
}
Optional<UnsignedLong> sequenceNumber = getNextUnsignedLong();
if (!sequenceNumber.isPresent()) {
return getTransformedValue(currentKey);
}
Optional<Integer> type = getNextNumberInStream();
if (!type.isPresent()) {
return getTransformedValue(currentKey);
}
Optional<byte[]> value = getNextByteArray();
value.ifPresent(bytes -> nextKey = Optional.of(
new KeyValue(key.get(), sequenceNumber.get(), type.get(), bytes)));
return getTransformedValue(currentKey);
} catch (IOException e) {
throw new RuntimeIOException(e);
}
```
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.ArgumentsProvider;
+import org.junit.jupiter.params.provider.ArgumentsSource;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue> iterator =
+ new ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue>(
+ tool, file.getAbsolutePath(), new ManagedOptions()) {
+
+ @Override
+ protected KeyValue getTransformedValue(Optional<KeyValue> value) {
+ return value.orElse(null);
+ }
+ }) {
+ while (iterator.hasNext()) {
+ ManagedSSTDumpIterator.KeyValue r = iterator.next();
+ Pair<String, Integer> recordKey = Pair.of(new String(r.getKey(),
+ StandardCharsets.UTF_8), r.getType());
+ Assertions.assertTrue(keys.containsKey(recordKey));
+ Assertions.assertEquals(
+ Optional.ofNullable(keys.get(recordKey)).orElse(""),
+ new String(r.getValue(), StandardCharsets.UTF_8));
+ keys.remove(recordKey);
+ }
+ Assertions.assertEquals(0, keys.size());
+ }
+ executorService.shutdown();
+ }
+ }
+
+ @Native("Managed Rocks Tools")
+ @ParameterizedTest
+ @ArgumentsSource(KeyValueFormatArgumentProvider.class)
+ public void testSSTDumpIteratorWithKeyFormat(String keyFormat,
+ String valueFormat) throws Exception {
+ TreeMap<Pair<String, Integer>, String> keys =
+ IntStream.range(0, 100).boxed().collect(
+ Collectors.toMap(
+ i -> Pair.of(String.format(keyFormat, i), i % 2),
+ i -> i % 2 == 0 ? "" : String.format(valueFormat, i),
+ (v1, v2) -> v2,
+ TreeMap::new));
+ testSSTDumpIteratorWithKeys(keys);
+ }
+}
+
+class KeyValueFormatArgumentProvider implements ArgumentsProvider {
+ @Override
+ public Stream<? extends Arguments> provideArguments(
+ ExtensionContext context) {
+ return Stream.of(
+ Arguments.of("'key%1$d=>", "%1$dvalue'"),
Review Comment:
nit: You can also add description of the test similar to [this](https://github.com/apache/ozone/blob/master/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestLDBCli.java#L119) or [this](https://github.com/apache/ozone/blob/3858cd102bbc3c328c24261d929c6549f5358a40/hadoop-hdds/rocksdb-checkpoint-differ/src/test/java/org/apache/ozone/rocksdiff/TestRocksDBCheckpointDiffer.java#L195) so that reader will know what you are testing from the description.
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/NativeLibraryLoader.java:
##########
@@ -132,15 +134,17 @@ private Optional<File> copyResourceFromJarToTemp(final String libraryName)
}
// create a temporary file to copy the library to
- final File temp = File.createTempFile(libraryName, getLibOsSuffix());
+ final File temp = File.createTempFile(libraryName, getLibOsSuffix(),
+ new File(""));
Review Comment:
Should this be current dir or some root dir instead of `""`? I don't know empty means current?
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -244,40 +221,45 @@ protected void finalize() throws Throwable {
* Class containing Parsed KeyValue Record from Sst Dumptool output.
*/
public static final class KeyValue {
- private String key;
- private Integer sequence;
+ private byte[] key;
+ private UnsignedLong sequence;
private Integer type;
+ private byte[] value;
- private String value;
-
- private KeyValue(String key, String sequence, String type,
- String value) {
+ private KeyValue(byte[] key, UnsignedLong sequence, Integer type,
+ byte[] value) {
this.key = key;
- this.sequence = Integer.valueOf(sequence);
- this.type = Integer.valueOf(type);
+ this.sequence = sequence;
+ this.type = type;
this.value = value;
}
- public String getKey() {
+ @SuppressFBWarnings("EI_EXPOSE_REP")
+ public byte[] getKey() {
return key;
}
- public Integer getSequence() {
+ public UnsignedLong getSequence() {
return sequence;
}
public Integer getType() {
return type;
}
- public String getValue() {
+ @SuppressFBWarnings("EI_EXPOSE_REP")
+ public byte[] getValue() {
return value;
}
@Override
public String toString() {
- return "KeyValue{" + "key='" + key + '\'' + ", sequence=" + sequence +
- ", type=" + type + ", value='" + value + '\'' + '}';
+ return "KeyValue{" +
+ "key=" + Arrays.toString(key) +
+ ", sequence=" + sequence +
+ ", type=" + type +
+ ", value=" + Arrays.toString(value) +
Review Comment:
Same as above.
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -180,33 +169,25 @@ public boolean hasNext() {
public T next() {
checkSanityOfProcess();
currentKey = nextKey;
- nextKey = null;
- boolean keyFound = false;
- while (!keyFound) {
- try {
- Optional<Integer> keyLength = getNextNumberInStream();
- if (!keyLength.isPresent()) {
- return getTransformedValue(currentKey);
- }
- String keyStr = readNextNumberOfCharsFromStream(keyLength.get());
- Matcher matcher = PATTERN_MATCHER.matcher(keyStr);
- if (keyStr.length() == keyLength.get() && matcher.find()) {
- Optional<Integer> valueLength = getNextNumberInStream();
- if (valueLength.isPresent()) {
- String valueStr = readNextNumberOfCharsFromStream(
- valueLength.get());
- if (valueStr.length() == valueLength.get()) {
- keyFound = true;
- nextKey = new KeyValue(matcher.group(PATTERN_KEY_GROUP_NUMBER),
- matcher.group(PATTERN_SEQ_GROUP_NUMBER),
- matcher.group(PATTERN_TYPE_GROUP_NUMBER),
- valueStr);
+ nextKey = Optional.empty();
+ try {
Review Comment:
nit: next() has too much nesting and could be simplified to this.
```suggestion
try {
Optional<byte[]> key = getNextByteArray();
if (!key.isPresent()) {
return getTransformedValue(currentKey);
}
Optional<UnsignedLong> sequenceNumber = getNextUnsignedLong();
if (!sequenceNumber.isPresent()) {
return getTransformedValue(currentKey);
}
Optional<Integer> type = getNextNumberInStream();
if (!type.isPresent()) {
return getTransformedValue(currentKey);
}
Optional<byte[]> value = getNextByteArray();
value.ifPresent(bytes -> nextKey = Optional.of(
new KeyValue(key.get(), sequenceNumber.get(), type.get(), bytes)));
return getTransformedValue(currentKey);
} catch (IOException e) {
throw new RuntimeIOException(e);
}
```
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -86,51 +79,47 @@ public ManagedSSTDumpIterator(ManagedSSTDumpTool sstDumpTool,
* @return Optional of the integer empty if no integer exists
*/
private Optional<Integer> getNextNumberInStream() throws IOException {
- StringBuilder value = new StringBuilder();
- int val;
- while ((val = processOutput.read()) != -1) {
- if (val >= '0' && val <= '9') {
- value.append((char) val);
- } else if (value.length() > 0) {
- break;
- }
+ int n = processOutput.read(intBuffer, 0, 4);
+ if (n == 4) {
Review Comment:
Is it considered as data corruption if bytes read are in between 1-3?
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/PipeInputStream.java:
##########
@@ -65,11 +65,13 @@ public int read() {
this.close();
return -1;
}
- if (numberOfBytesLeftToRead == 0) {
+ while (numberOfBytesLeftToRead == 0) {
numberOfBytesLeftToRead = readInternal(byteBuffer, capacity,
nativeHandle);
index = 0;
- return read();
+ if (numberOfBytesLeftToRead != 0) {
Review Comment:
Will `readInternal()` return -1 when there is not more bytes to read?
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -244,40 +221,45 @@ protected void finalize() throws Throwable {
* Class containing Parsed KeyValue Record from Sst Dumptool output.
*/
public static final class KeyValue {
- private String key;
- private Integer sequence;
+ private byte[] key;
+ private UnsignedLong sequence;
private Integer type;
+ private byte[] value;
Review Comment:
Make all fields final.
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -244,40 +221,45 @@ protected void finalize() throws Throwable {
* Class containing Parsed KeyValue Record from Sst Dumptool output.
*/
public static final class KeyValue {
- private String key;
- private Integer sequence;
+ private byte[] key;
+ private UnsignedLong sequence;
private Integer type;
+ private byte[] value;
- private String value;
-
- private KeyValue(String key, String sequence, String type,
- String value) {
+ private KeyValue(byte[] key, UnsignedLong sequence, Integer type,
+ byte[] value) {
this.key = key;
- this.sequence = Integer.valueOf(sequence);
- this.type = Integer.valueOf(type);
+ this.sequence = sequence;
+ this.type = type;
this.value = value;
}
- public String getKey() {
+ @SuppressFBWarnings("EI_EXPOSE_REP")
+ public byte[] getKey() {
return key;
}
- public Integer getSequence() {
+ public UnsignedLong getSequence() {
return sequence;
}
public Integer getType() {
return type;
}
- public String getValue() {
+ @SuppressFBWarnings("EI_EXPOSE_REP")
+ public byte[] getValue() {
return value;
}
@Override
public String toString() {
- return "KeyValue{" + "key='" + key + '\'' + ", sequence=" + sequence +
- ", type=" + type + ", value='" + value + '\'' + '}';
+ return "KeyValue{" +
+ "key=" + Arrays.toString(key) +
Review Comment:
I think `StringUtils.bytes2String(key)` should be use to make it human readable.
```suggestion
"key=" + StringUtils.bytes2String(key) +
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191803184
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -244,40 +221,45 @@ protected void finalize() throws Throwable {
* Class containing Parsed KeyValue Record from Sst Dumptool output.
*/
public static final class KeyValue {
- private String key;
- private Integer sequence;
+ private byte[] key;
+ private UnsignedLong sequence;
private Integer type;
+ private byte[] value;
- private String value;
-
- private KeyValue(String key, String sequence, String type,
- String value) {
+ private KeyValue(byte[] key, UnsignedLong sequence, Integer type,
+ byte[] value) {
this.key = key;
- this.sequence = Integer.valueOf(sequence);
- this.type = Integer.valueOf(type);
+ this.sequence = sequence;
+ this.type = type;
this.value = value;
}
- public String getKey() {
+ @SuppressFBWarnings("EI_EXPOSE_REP")
+ public byte[] getKey() {
return key;
}
- public Integer getSequence() {
+ public UnsignedLong getSequence() {
return sequence;
}
public Integer getType() {
return type;
}
- public String getValue() {
+ @SuppressFBWarnings("EI_EXPOSE_REP")
+ public byte[] getValue() {
return value;
}
@Override
public String toString() {
- return "KeyValue{" + "key='" + key + '\'' + ", sequence=" + sequence +
- ", type=" + type + ", value='" + value + '\'' + '}';
+ return "KeyValue{" +
+ "key=" + Arrays.toString(key) +
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1191803388
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -180,33 +169,25 @@ public boolean hasNext() {
public T next() {
checkSanityOfProcess();
currentKey = nextKey;
- nextKey = null;
- boolean keyFound = false;
- while (!keyFound) {
- try {
- Optional<Integer> keyLength = getNextNumberInStream();
- if (!keyLength.isPresent()) {
- return getTransformedValue(currentKey);
- }
- String keyStr = readNextNumberOfCharsFromStream(keyLength.get());
- Matcher matcher = PATTERN_MATCHER.matcher(keyStr);
- if (keyStr.length() == keyLength.get() && matcher.find()) {
- Optional<Integer> valueLength = getNextNumberInStream();
- if (valueLength.isPresent()) {
- String valueStr = readNextNumberOfCharsFromStream(
- valueLength.get());
- if (valueStr.length() == valueLength.get()) {
- keyFound = true;
- nextKey = new KeyValue(matcher.group(PATTERN_KEY_GROUP_NUMBER),
- matcher.group(PATTERN_SEQ_GROUP_NUMBER),
- matcher.group(PATTERN_TYPE_GROUP_NUMBER),
- valueStr);
+ nextKey = Optional.empty();
+ try {
Review Comment:
done
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -244,40 +221,45 @@ protected void finalize() throws Throwable {
* Class containing Parsed KeyValue Record from Sst Dumptool output.
*/
public static final class KeyValue {
- private String key;
- private Integer sequence;
+ private byte[] key;
+ private UnsignedLong sequence;
private Integer type;
+ private byte[] value;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1192637301
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedOptions options = new ManagedOptions();
Review Comment:
Please avoid nested try. Please change it to this way.
```
try {
// First try
}
try {
// Second try
}
```
Or extract out the code to separate functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195497841
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils.db.managed;
+
+import com.google.common.primitives.Bytes;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Named;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Matchers;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+/**
+ * Test for ManagedSSTDumpIterator.
+ */
+public class TestManagedSSTDumpIterator {
+
+ private void testSSTDumpIteratorWithKeys(
+ TreeMap<Pair<String, Integer>, String> keys) throws Exception {
+ File file = File.createTempFile("tmp_sst_file", ".sst");
+ file.deleteOnExit();
+ try (ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new ManagedSstFileWriter(
+ envOptions, managedOptions)) {
+ sstFileWriter.open(file.getAbsolutePath());
+ for (Map.Entry<Pair<String, Integer>, String> entry : keys.entrySet()) {
+ if (entry.getKey().getValue() == 0) {
+ sstFileWriter.delete(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8));
+ } else {
+ sstFileWriter.put(entry.getKey().getKey()
+ .getBytes(StandardCharsets.UTF_8),
+ entry.getValue().getBytes(StandardCharsets.UTF_8));
+ }
+ }
+ sstFileWriter.finish();
+ sstFileWriter.close();
+ ExecutorService executorService =
+ new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(1),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+ ManagedSSTDumpTool tool = new ManagedSSTDumpTool(executorService, 8192);
+ try (ManagedOptions options = new ManagedOptions();
+ ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue> iterator =
+ new ManagedSSTDumpIterator<ManagedSSTDumpIterator.KeyValue>(tool,
+ file.getAbsolutePath(), options) {
+
+ @Override
+ protected KeyValue getTransformedValue(Optional<KeyValue> value) {
+ return value.orElse(null);
+ }
+ }) {
+ while (iterator.hasNext()) {
+ ManagedSSTDumpIterator.KeyValue r = iterator.next();
+ Pair<String, Integer> recordKey = Pair.of(new String(r.getKey(),
+ StandardCharsets.UTF_8), r.getType());
+ Assertions.assertTrue(keys.containsKey(recordKey));
+ Assertions.assertEquals(
+ Optional.ofNullable(keys.get(recordKey)).orElse(""),
+ new String(r.getValue(), StandardCharsets.UTF_8));
+ keys.remove(recordKey);
+ }
+ Assertions.assertEquals(0, keys.size());
+ }
+ executorService.shutdown();
+ }
+ }
+
+ private static Stream<? extends Arguments> keyValueFormatArgs() {
+ return Stream.of(
+ Arguments.of(
+ Named.of("Key starting with a single quote",
+ "'key%1$d=>"),
+ Named.of("Value starting with a number ending with a" +
+ " single quote", "%1$dvalue'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number", "%1$dvalue%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")),
+ Arguments.of(
+ Named.of("Key starting with a single quote & ending" +
+ " with a number", "'key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$d'value%1$d'")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number", "key%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& containing null character & new line character",
+ "%1$dvalue\n\0%1$d")
+ ),
+ Arguments.of(
+ Named.of("Key ending with a number & containing" +
+ " a null character", "key\0%1$d"),
+ Named.of("Value starting & ending with a number " +
+ "& elosed within quotes", "%1$dvalue\r%1$d")
+ )
+ );
+ }
+
+ private static byte[] getBytes(Integer val) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1195737567
##########
hadoop-hdds/rocks-native/src/main/patches/rocks-native.patch:
##########
@@ -153,36 +144,83 @@ index eefbaaeee..503ec7aa6 100644
}
return Status::OK();
-@@ -478,21 +481,26 @@ Status SstFileDumper::ReadSequential(bool print_kv, uint64_t read_num,
+ }
+
++void write(int value, FILE* file) {
++ char b[4];
++ b[3] = value & 0x000000ff;
++ b[2] = (value & 0x0000ff00) >> 8;
++ b[1] = (value & 0x00ff0000) >> 16;
++ b[0] = (value & 0xff000000) >> 24;
++ std::fwrite(b, 4, 1, file);
++}
++
++void write(const char* value, int length, FILE* file) {
++ write(length, file);
++ fwrite(value, length, 1, file);
++}
++
++void write(const std::string& value, FILE* file) {
++ write(value.data(), (int)value.length(), file);
++
++}
++
++void write(Slice &slice, FILE* file) {
++ int size = (int)slice.size();
++ write(slice.data(), size, file);
++}
++
++void write(SequenceNumber sequenceNumber, FILE* file) {
++
++ char b[8];
++ int idx = 7;
++ while (idx >= 0) {
++ b[idx] = sequenceNumber % 256;
++ sequenceNumber/=256;
++ idx -= 1;
++ }
++ fwrite(b, 8, 1, file);
++}
++
++void write(ParsedInternalKey &key, FILE* file) {
++ write(key.user_key, file);
++ write(key.sequence, file);
++ write(static_cast<int>(key.type), file);
++}
Review Comment:
inconsistent indentation
```suggestion
+void write(ParsedInternalKey &key, FILE* file) {
+ write(key.user_key, file);
+ write(key.sequence, file);
+ write(static_cast<int>(key.type), file);
+}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #4584:
URL: https://github.com/apache/ozone/pull/4584#issuecomment-1550450178
Note to self and other committers: need to push new rocks-native SNAPSHOT version jar to Apache maven repo once this is merged to avoid long CI build time. Ref: https://github.com/apache/ratis/blob/master/DEPLOY.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1192680334
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -244,40 +245,46 @@ protected void finalize() throws Throwable {
* Class containing Parsed KeyValue Record from Sst Dumptool output.
*/
public static final class KeyValue {
- private String key;
- private Integer sequence;
- private Integer type;
- private String value;
+ private final byte[] key;
+ private final UnsignedLong sequence;
+ private final Integer type;
+ private final byte[] value;
- private KeyValue(String key, String sequence, String type,
- String value) {
+ private KeyValue(byte[] key, UnsignedLong sequence, Integer type,
+ byte[] value) {
this.key = key;
- this.sequence = Integer.valueOf(sequence);
- this.type = Integer.valueOf(type);
+ this.sequence = sequence;
+ this.type = type;
this.value = value;
}
- public String getKey() {
+ @SuppressFBWarnings("EI_EXPOSE_REP")
+ public byte[] getKey() {
return key;
}
- public Integer getSequence() {
+ public UnsignedLong getSequence() {
return sequence;
}
public Integer getType() {
return type;
}
- public String getValue() {
+ @SuppressFBWarnings("EI_EXPOSE_REP")
+ public byte[] getValue() {
return value;
}
@Override
public String toString() {
- return "KeyValue{" + "key='" + key + '\'' + ", sequence=" + sequence +
- ", type=" + type + ", value='" + value + '\'' + '}';
+ return "KeyValue{" +
+ "key=" + StringUtils.bytes2String(key) +
Review Comment:
no it cannot be null. We throw illegalstateexception even before it comes here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4584: HDDS-8306. [Snapshot] Add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4584:
URL: https://github.com/apache/ozone/pull/4584#discussion_r1185761221
##########
hadoop-hdds/rocks-native/pom.xml:
##########
@@ -436,5 +449,24 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>noNativeTest</id>
Review Comment:
[Conventionally](https://maven.apache.org/guides/introduction/introduction-to-profiles.html#profile-order) maven profile id uses hyphen with all lower cases:
```suggestion
<id>disable-native-testing</id>
```
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestManagedSSTDumpIterator.java:
##########
Review Comment:
Apache License header?
##########
hadoop-hdds/rocks-native/src/test/java/org/apache/hadoop/hdds/utils/db/managed/NativeTests.java:
##########
Review Comment:
Apache License header?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #4584:
URL: https://github.com/apache/ozone/pull/4584#issuecomment-1559959005
> > Note to self and other committers: need to push new rocks-native SNAPSHOT version jar to Apache maven repo once this is merged to avoid long CI build time. Ref: [apache/ratis@`master`/DEPLOY.md](https://github.com/apache/ratis/blob/master/DEPLOY.md?rgh-link-date=2023-05-16T22%3A41%3A19Z)
>
> @prashantpogde Have you pushed new SNAPSHOT artifact to Apache maven yet?
Yeah it has been pushed by @aswinshakil
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on pull request #4584: HDDS-8306. [Snapshot] Fix SSTDumptool to allow null character & add tests for SST dump tool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #4584:
URL: https://github.com/apache/ozone/pull/4584#issuecomment-1559917705
> Note to self and other committers: need to push new rocks-native SNAPSHOT version jar to Apache maven repo once this is merged to avoid long CI build time. Ref: [apache/ratis@`master`/DEPLOY.md](https://github.com/apache/ratis/blob/master/DEPLOY.md?rgh-link-date=2023-05-16T22%3A41%3A19Z)
@prashantpogde Have you pushed new SNAPSHOT artifact to Apache maven yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org