You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/07/16 09:25:21 UTC
[GitHub] [hadoop-ozone] sadanand48 opened a new pull request #1206: HDDS-3972. Add option to limit number of items displaying through ldb tool.
sadanand48 opened a new pull request #1206:
URL: https://github.com/apache/hadoop-ozone/pull/1206
## What changes were proposed in this pull request?
The change here is to add an option to the ldb tool scan command to limit number of displayed items.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3972
## How was this patch tested?
Added Unit Tests for the tool.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] sadanand48 commented on pull request #1206: HDDS-3972. Add option to limit number of items displaying through ldb tool.
Posted by GitBox <gi...@apache.org>.
sadanand48 commented on pull request #1206:
URL: https://github.com/apache/hadoop-ozone/pull/1206#issuecomment-665681118
Thanks @elek for the review. Addressed Review Comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #1206: HDDS-3972. Add option to limit number of items displaying through ldb tool.
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #1206:
URL: https://github.com/apache/hadoop-ozone/pull/1206#discussion_r460753487
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmLDBCli.java
##########
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.hdds.HddsConfigKeys;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.TestDataUtil;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.debug.DBScanner;
+import org.apache.hadoop.ozone.debug.RDBParser;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_DB_NAME;
+
+/**
+ * This class tests the Debug LDB CLI that reads from an om.db file.
+ */
+public class TestOmLDBCli {
+ private MiniOzoneCluster cluster = null;
+ private OzoneConfiguration conf;
+
+ private RDBParser rdbParser;
+ private DBScanner dbScanner;
+
+ @Before
+ public void setup() throws Exception {
+ conf = new OzoneConfiguration();
+ cluster = MiniOzoneCluster.newBuilder(conf).build();
Review comment:
`MiniOzoneCluster` is very heavy weight for this kind of testing. It's very slow (requires to start all the services) and easier to be flaky.
I would suggest to use pure rocksdb creation here (for example use pure `DBStoreBuilder` and create a pure rocksdb dir). Or if you prefer to test SCM specific rocksdb, you can use `SCMMetadataStoreImpl`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] sadanand48 commented on a change in pull request #1206: HDDS-3972. Add option to limit number of items displaying through ldb tool.
Posted by GitBox <gi...@apache.org>.
sadanand48 commented on a change in pull request #1206:
URL: https://github.com/apache/hadoop-ozone/pull/1206#discussion_r462321624
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmLDBCli.java
##########
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.hdds.HddsConfigKeys;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.TestDataUtil;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.debug.DBScanner;
+import org.apache.hadoop.ozone.debug.RDBParser;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_DB_NAME;
+
+/**
+ * This class tests the Debug LDB CLI that reads from an om.db file.
+ */
+public class TestOmLDBCli {
+ private MiniOzoneCluster cluster = null;
+ private OzoneConfiguration conf;
+
+ private RDBParser rdbParser;
+ private DBScanner dbScanner;
+
+ @Before
+ public void setup() throws Exception {
+ conf = new OzoneConfiguration();
+ cluster = MiniOzoneCluster.newBuilder(conf).build();
Review comment:
I have used DBStoreBuilder now. But i had to create a db with the same name 'om.db'. as my tool reads on from those DB whose DBDefinition is defined. I'm not sure whether that would be a problem.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmLDBCli.java
##########
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.hdds.HddsConfigKeys;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.TestDataUtil;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.debug.DBScanner;
+import org.apache.hadoop.ozone.debug.RDBParser;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_DB_NAME;
+
+/**
+ * This class tests the Debug LDB CLI that reads from an om.db file.
+ */
+public class TestOmLDBCli {
+ private MiniOzoneCluster cluster = null;
+ private OzoneConfiguration conf;
+
+ private RDBParser rdbParser;
+ private DBScanner dbScanner;
+
+ @Before
+ public void setup() throws Exception {
+ conf = new OzoneConfiguration();
+ cluster = MiniOzoneCluster.newBuilder(conf).build();
Review comment:
I have used DBStoreBuilder now. But i had to create a db with the same name 'om.db' as my tool reads on from those DB whose DBDefinition is defined. I'm not sure whether that would be a problem.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmLDBCli.java
##########
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.hdds.HddsConfigKeys;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.TestDataUtil;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.debug.DBScanner;
+import org.apache.hadoop.ozone.debug.RDBParser;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_DB_NAME;
+
+/**
+ * This class tests the Debug LDB CLI that reads from an om.db file.
+ */
+public class TestOmLDBCli {
+ private MiniOzoneCluster cluster = null;
+ private OzoneConfiguration conf;
+
+ private RDBParser rdbParser;
+ private DBScanner dbScanner;
+
+ @Before
+ public void setup() throws Exception {
+ conf = new OzoneConfiguration();
+ cluster = MiniOzoneCluster.newBuilder(conf).build();
Review comment:
I have used DBStoreBuilder now. But i had to create a db with the same name 'om.db' as my tool reads only from those DB whose DBDefinition is defined. I'm not sure whether that would be a problem.
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmLDBCli.java
##########
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.hdds.HddsConfigKeys;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.TestDataUtil;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.debug.DBScanner;
+import org.apache.hadoop.ozone.debug.RDBParser;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_DB_NAME;
+
+/**
+ * This class tests the Debug LDB CLI that reads from an om.db file.
+ */
+public class TestOmLDBCli {
+ private MiniOzoneCluster cluster = null;
+ private OzoneConfiguration conf;
+
+ private RDBParser rdbParser;
+ private DBScanner dbScanner;
+
+ @Before
+ public void setup() throws Exception {
+ conf = new OzoneConfiguration();
+ cluster = MiniOzoneCluster.newBuilder(conf).build();
Review comment:
I have used DBStoreBuilder now. But i had to create a db with the same name 'om.db' as the tool only reads from those DB whose DBDefinition is defined. I'm not sure whether that would be a problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1206: HDDS-3972. Add option to limit number of items displaying through ldb tool.
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1206:
URL: https://github.com/apache/hadoop-ozone/pull/1206#issuecomment-666305595
Thanks the update @sadanand48. Yes it looks better (and significant better) for me.
I realized two other NITs in the test.
1. `@Test(expected = IllegalArgumentException.class)`
When you use complex test cases (which multiple assertions) it's not safe to use the `expected` attribute. Actually this test passes even if the method *always* returns with IllegalArgumentException as the remaining asserstions will be ignored in this case.
For example the last line, is ignored in the current patch.
```
Assert.assertEquals(0, keyNames.size());
```
It's never called.
I think it's more safe to check if that specific call throws an exception or not:
```
try {
getKeyNames(dbScanner, "keyTable", keyNames);
Assert.fail("Illegal argument exception is expected");
} catch (IllegalArgumentException ex) {
}
```
2. Is more minor, but the readability can be improved with using
```
List<String> keyNames = getKeyNames(dbScanner, "keyTable");
```
instead of
```
getKeyNames(dbScanner, "keyTable", keyNames);
```
I know that it's possible to pass `keyNames` as reference and reset and modify the content, but the behavior seems to be more obvious (easier to read) if you just return with the selected values.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek merged pull request #1206: HDDS-3972. Add option to limit number of items displaying through ldb tool.
Posted by GitBox <gi...@apache.org>.
elek merged pull request #1206:
URL: https://github.com/apache/hadoop-ozone/pull/1206
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org