You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/09/22 23:54:21 UTC
[GitHub] [ozone] bharatviswa504 opened a new pull request #2675: HDDS-5777. Provide an option to dump table scan data to file.
bharatviswa504 opened a new pull request #2675:
URL: https://github.com/apache/ozone/pull/2675
## What changes were proposed in this pull request?
Provide an option to dump entire table column family data to a file.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5777
## How was this patch tested?
Manually verified it on the cluster
```
ozone debug ldb --db=/var/lib/hadoop-ozone/scm/data/scm.db/ scan --column_family=containers --with-keys --out=/tmp/containers1 -l=-1
less /tmp/containers1
{
"id": 1
} -> {
"state": "OPEN",
"pipelineID": {
"id": "054f8658-81bf-4e81-a71e-4928130f5993"
},
"replicationConfig": {
"replicationFactor": "THREE"
},
"usedBytes": 0,
"numberOfKeys": 0,
"lastUsed": {
"seconds": 1632353739,
"nanos": 455000000
},
"stateEnterTime": {
"seconds": 1632337329,
"nanos": 692000000
},
"owner": "om1",
"containerID": 1,
"deleteTransactionId": 0,
"sequenceId": 0
}
{
"id": 2
} -> {
"state": "OPEN",
"pipelineID": {
"id": "054f8658-81bf-4e81-a71e-4928130f5993"
},
"replicationConfig": {
"replicationFactor": "THREE"
},
"usedBytes": 0,
"numberOfKeys": 0,
"lastUsed": {
"seconds": 1632353739,
"nanos": 471000000
},
"stateEnterTime": {
"seconds": 1632337343,
"nanos": 998000000
},
"owner": "om1",
"containerID": 2,
"deleteTransactionId": 0,
"sequenceId": 0
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2675: HDDS-5777. Provide an option to dump table scan data to file.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2675:
URL: https://github.com/apache/ozone/pull/2675#issuecomment-928227735
Fixed tests and added a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2675: HDDS-5777. Provide an option to dump table scan data to file.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2675:
URL: https://github.com/apache/ozone/pull/2675#discussion_r717021815
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmLDBCli.java
##########
@@ -98,15 +103,70 @@ public void testOMDB() throws Exception {
Assert.assertTrue(getKeyNames(dbScanner).contains("key1"));
Assert.assertTrue(getKeyNames(dbScanner).contains("key5"));
Assert.assertFalse(getKeyNames(dbScanner).contains("key6"));
+
DBScanner.setLimit(1);
Assert.assertEquals(1, getKeyNames(dbScanner).size());
- DBScanner.setLimit(-1);
+
+ DBScanner.setLimit(0);
try {
getKeyNames(dbScanner);
Assert.fail("IllegalArgumentException is expected");
}catch (IllegalArgumentException e){
//ignore
}
+
+ // If set with -1, check if it dumps entire table data.
+ DBScanner.setLimit(-1);
+ Assert.assertEquals(5, getKeyNames(dbScanner).size());
+
+ // Test dump to file.
+ String outFile = "/tmp/keyTable" + LocalDate.now();
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] avijayanhwx commented on a change in pull request #2675: HDDS-5777. Provide an option to dump table scan data to file.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #2675:
URL: https://github.com/apache/ozone/pull/2675#discussion_r717015280
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmLDBCli.java
##########
@@ -98,15 +103,70 @@ public void testOMDB() throws Exception {
Assert.assertTrue(getKeyNames(dbScanner).contains("key1"));
Assert.assertTrue(getKeyNames(dbScanner).contains("key5"));
Assert.assertFalse(getKeyNames(dbScanner).contains("key6"));
+
DBScanner.setLimit(1);
Assert.assertEquals(1, getKeyNames(dbScanner).size());
- DBScanner.setLimit(-1);
+
+ DBScanner.setLimit(0);
try {
getKeyNames(dbScanner);
Assert.fail("IllegalArgumentException is expected");
}catch (IllegalArgumentException e){
//ignore
}
+
+ // If set with -1, check if it dumps entire table data.
+ DBScanner.setLimit(-1);
+ Assert.assertEquals(5, getKeyNames(dbScanner).size());
+
+ // Test dump to file.
+ String outFile = "/tmp/keyTable" + LocalDate.now();
Review comment:
Nit. Maybe better to use Java tmp dir?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2675: HDDS-5777. Provide an option to dump table scan data to file.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2675:
URL: https://github.com/apache/ozone/pull/2675#discussion_r717021569
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmLDBCli.java
##########
@@ -98,15 +103,70 @@ public void testOMDB() throws Exception {
Assert.assertTrue(getKeyNames(dbScanner).contains("key1"));
Assert.assertTrue(getKeyNames(dbScanner).contains("key5"));
Assert.assertFalse(getKeyNames(dbScanner).contains("key6"));
+
DBScanner.setLimit(1);
Assert.assertEquals(1, getKeyNames(dbScanner).size());
- DBScanner.setLimit(-1);
+
+ DBScanner.setLimit(0);
try {
getKeyNames(dbScanner);
Assert.fail("IllegalArgumentException is expected");
}catch (IllegalArgumentException e){
//ignore
}
+
+ // If set with -1, check if it dumps entire table data.
+ DBScanner.setLimit(-1);
+ Assert.assertEquals(5, getKeyNames(dbScanner).size());
+
+ // Test dump to file.
+ String outFile = "/tmp/keyTable" + LocalDate.now();
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2675: HDDS-5777. Provide an option to dump table scan data to file.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2675:
URL: https://github.com/apache/ozone/pull/2675#discussion_r717021569
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmLDBCli.java
##########
@@ -98,15 +103,70 @@ public void testOMDB() throws Exception {
Assert.assertTrue(getKeyNames(dbScanner).contains("key1"));
Assert.assertTrue(getKeyNames(dbScanner).contains("key5"));
Assert.assertFalse(getKeyNames(dbScanner).contains("key6"));
+
DBScanner.setLimit(1);
Assert.assertEquals(1, getKeyNames(dbScanner).size());
- DBScanner.setLimit(-1);
+
+ DBScanner.setLimit(0);
try {
getKeyNames(dbScanner);
Assert.fail("IllegalArgumentException is expected");
}catch (IllegalArgumentException e){
//ignore
}
+
+ // If set with -1, check if it dumps entire table data.
+ DBScanner.setLimit(-1);
+ Assert.assertEquals(5, getKeyNames(dbScanner).size());
+
+ // Test dump to file.
+ String outFile = "/tmp/keyTable" + LocalDate.now();
Review comment:
Done
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmLDBCli.java
##########
@@ -98,15 +103,70 @@ public void testOMDB() throws Exception {
Assert.assertTrue(getKeyNames(dbScanner).contains("key1"));
Assert.assertTrue(getKeyNames(dbScanner).contains("key5"));
Assert.assertFalse(getKeyNames(dbScanner).contains("key6"));
+
DBScanner.setLimit(1);
Assert.assertEquals(1, getKeyNames(dbScanner).size());
- DBScanner.setLimit(-1);
+
+ DBScanner.setLimit(0);
try {
getKeyNames(dbScanner);
Assert.fail("IllegalArgumentException is expected");
}catch (IllegalArgumentException e){
//ignore
}
+
+ // If set with -1, check if it dumps entire table data.
+ DBScanner.setLimit(-1);
+ Assert.assertEquals(5, getKeyNames(dbScanner).size());
+
+ // Test dump to file.
+ String outFile = "/tmp/keyTable" + LocalDate.now();
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] avijayanhwx commented on pull request #2675: HDDS-5777. Provide an option to dump table scan data to file.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on pull request #2675:
URL: https://github.com/apache/ozone/pull/2675#issuecomment-928733909
Thank you for the contribution @bharatviswa504.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2675: HDDS-5777. Provide an option to dump table scan data to file.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2675:
URL: https://github.com/apache/ozone/pull/2675#issuecomment-928227735
Fixed tests and added a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] avijayanhwx commented on pull request #2675: HDDS-5777. Provide an option to dump table scan data to file.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on pull request #2675:
URL: https://github.com/apache/ozone/pull/2675#issuecomment-928733909
Thank you for the contribution @bharatviswa504.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] avijayanhwx merged pull request #2675: HDDS-5777. Provide an option to dump table scan data to file.
Posted by GitBox <gi...@apache.org>.
avijayanhwx merged pull request #2675:
URL: https://github.com/apache/ozone/pull/2675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] avijayanhwx merged pull request #2675: HDDS-5777. Provide an option to dump table scan data to file.
Posted by GitBox <gi...@apache.org>.
avijayanhwx merged pull request #2675:
URL: https://github.com/apache/ozone/pull/2675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] avijayanhwx commented on a change in pull request #2675: HDDS-5777. Provide an option to dump table scan data to file.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #2675:
URL: https://github.com/apache/ozone/pull/2675#discussion_r717015280
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmLDBCli.java
##########
@@ -98,15 +103,70 @@ public void testOMDB() throws Exception {
Assert.assertTrue(getKeyNames(dbScanner).contains("key1"));
Assert.assertTrue(getKeyNames(dbScanner).contains("key5"));
Assert.assertFalse(getKeyNames(dbScanner).contains("key6"));
+
DBScanner.setLimit(1);
Assert.assertEquals(1, getKeyNames(dbScanner).size());
- DBScanner.setLimit(-1);
+
+ DBScanner.setLimit(0);
try {
getKeyNames(dbScanner);
Assert.fail("IllegalArgumentException is expected");
}catch (IllegalArgumentException e){
//ignore
}
+
+ // If set with -1, check if it dumps entire table data.
+ DBScanner.setLimit(-1);
+ Assert.assertEquals(5, getKeyNames(dbScanner).size());
+
+ // Test dump to file.
+ String outFile = "/tmp/keyTable" + LocalDate.now();
Review comment:
Nit. Maybe better to use Java tmp dir?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org