You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/07/24 21:14:59 UTC

[GitHub] [hadoop-ozone] avijayanhwx opened a new pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

avijayanhwx opened a new pull request #1255:
URL: https://github.com/apache/hadoop-ozone/pull/1255


   ## What changes were proposed in this pull request?
   
   - ReconNodeManager uses node db in an old format which is not part of ReconDBDefinition. Move the definition to ReconDBDefinition.
   - Create DB Definition for Recon Container DB.
   - Modify DBScanner tool to allow it to read Recon DBs.
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-4021
   
   ## How was this patch tested?
   Manually tested on cluster.
   Added unit test for DBScanner changes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on pull request #1255:
URL: https://github.com/apache/hadoop-ozone/pull/1255#issuecomment-666542216


   > I am still not sure about this one (but as I wrote I am not familiar). In case of a new snapshot: why don't we drop the previous one and start to use a new one? Why do we need a new db with new timestamp? To have a backup? Or to use the old version for some reason?
   
   @elek We actually do delete the old Recon container Key DB entirely after creating a new one from new OM snapshot. We rely on File.delete() rather than RocksDB delete since it is easier & faster. The only reason we 'tag' the name of the DB with the timestamp is to help with troubleshooting Recon - OM sync problems. Also, it is easier to create new DB with new timestamp, and then delete old DB which has old timestamp in its name. Based on the DB name and the logs, we know when and why the DB was created. In the future, when Recon becomes mature, we can remove the timestamp in the DB.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] elek merged pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

Posted by GitBox <gi...@apache.org>.
elek merged pull request #1255:
URL: https://github.com/apache/hadoop-ozone/pull/1255


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #1255:
URL: https://github.com/apache/hadoop-ozone/pull/1255#discussion_r463151665



##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/DBScanner.java
##########
@@ -44,28 +44,46 @@
             description = "Table name")
   private String tableName;
 
+  @CommandLine.Option(names = {"--withKey"},

Review comment:
       Sure, will do.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] elek commented on pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1255:
URL: https://github.com/apache/hadoop-ozone/pull/1255#issuecomment-666323762


   > The container key DB is a by product of processing the OM snapshot DB. It hosts the reverse mapping of the OM data, the Container->Key mapping in a specific format
   
   Thanks to explain it. :bulb:
   
   Now I understand, if om is time-stampe-ed, we need a same internal db.
     
   > Based on whether we take a whole snapshot or just apply delta changes from the OM DB, we have to create a "new" container key DB or just add more entries to it. Hence, the "timestamp" is associated with its name
   
   I am still not sure about this one (but as I wrote I am not familiar). In case of a new snapshot: why don't we drop the previous one and start to use a new one? Why do we need a new db with new timestamp? To have a backup? Or to use the old version for some reason?
   
   (Sorry, it's not strongly related to this patch. And wouldn't like to challenge anything, just try to understand, for example to create a video when I understood ;-)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on pull request #1255:
URL: https://github.com/apache/hadoop-ozone/pull/1255#issuecomment-665693849


   @elek Thanks for looking at the patch.
   
   > I suppose they are separated because the 1 and 2 supposed to be a mirror of original SCM and OM dbs. OM db is updated 
   > with downloaded snapshots which (I guess) required to be a separated DB
   
   Yes, that is correct. The container key DB is a by product of processing the OM snapshot DB. It hosts the reverse mapping of the OM data, the Container->Key mapping in a specific format. Based on whether we take a whole snapshot or just apply delta changes from the OM DB, we have to create a "new" container key DB or just add more entries to it. Hence, the "timestamp" is associated with its name. In a steady state Recon, the container key DB will be created once, and only delta updates will be added to it. The same thing could also have been done by adding a new column family to the OM snapshot DB, and then just storing the reverse mapping there. But, to maintain separation of concerns, and possible scale issues in the future (since num_record(container key DB) == num_records(OM DB)), we had gone with this approach. We can definitely revisit this logic. 
   
   Moreover, in the future, the recon container key DB will likely be reused by Recon to manage all its LSM metadata, thereby limiting the number of RocksDBs in Recon to 3 (OM, SCM and Recon's internal). 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] elek commented on pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1255:
URL: https://github.com/apache/hadoop-ozone/pull/1255#issuecomment-666670426


   > The only reason we 'tag' the name of the DB with the timestamp is to help with troubleshooting Recon - OM sync problems...
   
   Oh, got it. Thanks to explain it. Sounds reasonable...
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] elek commented on a change in pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #1255:
URL: https://github.com/apache/hadoop-ozone/pull/1255#discussion_r462949586



##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/DBDefinitionFactory.java
##########
@@ -40,10 +46,24 @@ private DBDefinitionFactory() {
     dbMap.put(new OMDBDefinition().getName(), new OMDBDefinition());
   }
 
-  public static DBDefinition getDefinition(String dbName){
+  public static DBDefinition getDefinition(String dbName, String type){

Review comment:
       I am not sure if we need type here. I think this can be simplified as:
   
   ```java
    public static DBDefinition getDefinition(String dbName, String type){
       if (dbMap.containsKey(dbName)){
         return dbMap.get(dbName);
     }
       if (dbName.equals(RECON_SCM_DB_NAME)) {
         return new ReconSCMDBDefinition();
       } else if (dbName.startsWith(RECON_CONTAINER_KEY_DB)) {
         return new ReconDBDefinition(dbName);
       } else if (dbName.startsWith(RECON_OM_SNAPSHOT_DB)) {
         return new OMDBDefinition();
       }
       return null;
     }
   ```
   
   Seems to be a good logic which just works.
   
   Or actually `RECON_SCM_DB_NAME` also can be added to the map, so you don't need the firts condition

##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/DBScanner.java
##########
@@ -44,28 +44,46 @@
             description = "Table name")
   private String tableName;
 
+  @CommandLine.Option(names = {"--withKey"},

Review comment:
       NIT: as I am an oldschool command line user, I would prefer to use `--with-keys` instead of camel case




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #1255:
URL: https://github.com/apache/hadoop-ozone/pull/1255#discussion_r463151520



##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/DBDefinitionFactory.java
##########
@@ -40,10 +46,24 @@ private DBDefinitionFactory() {
     dbMap.put(new OMDBDefinition().getName(), new OMDBDefinition());
   }
 
-  public static DBDefinition getDefinition(String dbName){
+  public static DBDefinition getDefinition(String dbName, String type){

Review comment:
       You are right. At the start of writing this API, I was convinced I need a 'type' parameter. At the end of my patch, that was no longer true :) Forgot to remove this assumption. I will fix it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org