You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "InvisibleProgrammer (via GitHub)" <gi...@apache.org> on 2024/02/14 10:58:29 UTC

[PR] HIVE-28078: TestTxnDbUtil should generate csv files when we query the metastore database [hive]

InvisibleProgrammer opened a new pull request, #5085:
URL: https://github.com/apache/hive/pull/5085

   queryToString separates the columns with 3 spaces. It would improve the debugging experience if we have a method that shows the data in CSV format so that the output can be easily imported into a spreadsheet.
   
   ### What changes were proposed in this pull request?
   Two new method in TestTxnDbUtil: queryToCsv (with or without header). 
   
   ### Why are the changes needed?
   To easily be able to import the output into a sheet during a debugging session.
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### Is the change a dependency upgrade?
   No
   
   ### How was this patch tested?
   Manually
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-28078: TestTxnDbUtil should generate csv files when we query the metastore database [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5085:
URL: https://github.com/apache/hive/pull/5085#discussion_r1499522716


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestTxnDbUtil.java:
##########
@@ -338,10 +338,24 @@ public static int countQueryAgent(Configuration conf, String countQuery) throws
       closeResources(conn, stmt, rs);
     }
   }
+
   public static String queryToString(Configuration conf, String query) throws Exception {
     return queryToString(conf, query, true);
   }
-  public static String queryToString(Configuration conf, String query, boolean includeHeader)
+
+  public static String queryToString(Configuration conf, String query, boolean includeHeader) throws Exception {
+    return queryToString(conf, query, includeHeader, "   ");
+  }
+
+  public static String queryToCsv(Configuration conf, String query) throws Exception {

Review Comment:
   okay, it's for debugging purpose, it's fine to have it this way



##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestTxnDbUtil.java:
##########
@@ -338,10 +338,24 @@ public static int countQueryAgent(Configuration conf, String countQuery) throws
       closeResources(conn, stmt, rs);
     }
   }
+
   public static String queryToString(Configuration conf, String query) throws Exception {
     return queryToString(conf, query, true);
   }
-  public static String queryToString(Configuration conf, String query, boolean includeHeader)
+
+  public static String queryToString(Configuration conf, String query, boolean includeHeader) throws Exception {
+    return queryToString(conf, query, includeHeader, "   ");
+  }
+
+  public static String queryToCsv(Configuration conf, String query) throws Exception {

Review Comment:
   okay,got is, it's for debugging purposes, it's fine to have it this way



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-28078: TestTxnDbUtil should generate csv files when we query the metastore database [hive]

Posted by "InvisibleProgrammer (via GitHub)" <gi...@apache.org>.
InvisibleProgrammer commented on code in PR #5085:
URL: https://github.com/apache/hive/pull/5085#discussion_r1499112278


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestTxnDbUtil.java:
##########
@@ -338,10 +338,24 @@ public static int countQueryAgent(Configuration conf, String countQuery) throws
       closeResources(conn, stmt, rs);
     }
   }
+
   public static String queryToString(Configuration conf, String query) throws Exception {
     return queryToString(conf, query, true);
   }
-  public static String queryToString(Configuration conf, String query, boolean includeHeader)
+
+  public static String queryToString(Configuration conf, String query, boolean includeHeader) throws Exception {
+    return queryToString(conf, query, includeHeader, "   ");
+  }
+
+  public static String queryToCsv(Configuration conf, String query) throws Exception {

Review Comment:
   > what are the callers of this new queryToCsv method? I cannot see it used
   
   As the summary says, the intention is to improve debugging experience. 
   With the current queryToString method, if I want to analyze the output, my only choice if I want a header or not. But it separates the columns with 4 spaces. That is far from any usual. 
   So in case I want to use it in a spreadsheet app, I need to manually convert 4 spaces to a separator character. After that, I can import it into a spreadsheet.
   With that method, we can do it in one step. 
   
   > also, basically, I like that you introduced new convenience util methods, however, I cannot see their value added to the original queryToString this time: what if I would like to create a CSV with ";" separator? currently, I should call queryToString with columnSeparator=";" despite the new util methods
   so I believe that queryToCsv doesn't add more value, it just expresses an assumption that CSV is always written with comma separators (I know, CSV means comma separated, but still, I can see semicolon-separated stuff in many places)
   
   In case you want to use a non-standard characther, like `;`, you can use the modified method that accepts the separator character. I choosed `,` as that is the original idea of CSV: to separate with colons. 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-28078: TestTxnDbUtil should generate csv files when we query the metastore database [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5085:
URL: https://github.com/apache/hive/pull/5085#issuecomment-1943735933

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5085) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5085&resolved=false&inNewCodePeriod=true)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5085&resolved=false&inNewCodePeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5085)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-28078: TestTxnDbUtil should generate csv files when we query the metastore database [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5085:
URL: https://github.com/apache/hive/pull/5085#discussion_r1499070972


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestTxnDbUtil.java:
##########
@@ -338,10 +338,24 @@ public static int countQueryAgent(Configuration conf, String countQuery) throws
       closeResources(conn, stmt, rs);
     }
   }
+
   public static String queryToString(Configuration conf, String query) throws Exception {
     return queryToString(conf, query, true);
   }
-  public static String queryToString(Configuration conf, String query, boolean includeHeader)
+
+  public static String queryToString(Configuration conf, String query, boolean includeHeader) throws Exception {
+    return queryToString(conf, query, includeHeader, "   ");
+  }
+
+  public static String queryToCsv(Configuration conf, String query) throws Exception {

Review Comment:
   what are the callers of this new queryToCsv method? I cannot see it used
   
   also, basically, I like that you introduced new convenience util methods, however, I cannot see their value added to the original queryToString this time: what if I would like to create a CSV with ";" separator? currently, I should call queryToString with columnSeparator=";" despite the new util methods
   so I believe that queryToCsv doesn't add more value, it just expresses an assumption that CSV is always written with comma separators (I know, CSV means comma separated, but still, I can see semicolon-separated stuff in many places)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-28078: TestTxnDbUtil should generate csv files when we query the metastore database [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog merged PR #5085:
URL: https://github.com/apache/hive/pull/5085


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-28078: TestTxnDbUtil should generate csv files when we query the metastore database [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5085:
URL: https://github.com/apache/hive/pull/5085#discussion_r1499070972


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestTxnDbUtil.java:
##########
@@ -338,10 +338,24 @@ public static int countQueryAgent(Configuration conf, String countQuery) throws
       closeResources(conn, stmt, rs);
     }
   }
+
   public static String queryToString(Configuration conf, String query) throws Exception {
     return queryToString(conf, query, true);
   }
-  public static String queryToString(Configuration conf, String query, boolean includeHeader)
+
+  public static String queryToString(Configuration conf, String query, boolean includeHeader) throws Exception {
+    return queryToString(conf, query, includeHeader, "   ");
+  }
+
+  public static String queryToCsv(Configuration conf, String query) throws Exception {

Review Comment:
   what are the callers of this new queryToCsv method? I cannot see it used
   
   also, basically, I like that you introduced new convenience util methods, however, I cannot see their value added to the original queryToString: what if I would like to create a CSV with ";" separator? currently, I should call queryToString with columnSeparator=";"
   so I believe that queryToCsv doesn't add more value, it just expresses an assumption that CSV is always written with comma separators (I know, CSV means comma separated, but still, I can see semicolon-separated stuff in many places)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-28078: TestTxnDbUtil should generate csv files when we query the metastore database [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5085:
URL: https://github.com/apache/hive/pull/5085#discussion_r1489421947


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestTxnDbUtil.java:
##########
@@ -370,6 +384,8 @@ public static String queryToString(Configuration conf, String query, boolean inc
     return sb.toString();
   }
 
+

Review Comment:
   nit: is this needed? we already have 1 line space between the methods



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-28078: TestTxnDbUtil should generate csv files when we query the metastore database [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5085:
URL: https://github.com/apache/hive/pull/5085#discussion_r1499070972


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestTxnDbUtil.java:
##########
@@ -338,10 +338,24 @@ public static int countQueryAgent(Configuration conf, String countQuery) throws
       closeResources(conn, stmt, rs);
     }
   }
+
   public static String queryToString(Configuration conf, String query) throws Exception {
     return queryToString(conf, query, true);
   }
-  public static String queryToString(Configuration conf, String query, boolean includeHeader)
+
+  public static String queryToString(Configuration conf, String query, boolean includeHeader) throws Exception {
+    return queryToString(conf, query, includeHeader, "   ");
+  }
+
+  public static String queryToCsv(Configuration conf, String query) throws Exception {

Review Comment:
   what are the callers of this new queryToCsv method? I cannot see it used
   
   also, basically, I like that you introduced new convenience util methods, however, I cannot see their value added to the original queryToString: what if I would like to create a CSV with ";" separator? currently, I should call queryToString with columnSeparator=";"
   so I believe that queryToCsv doesn't add more value, it just express an assumption that CSV is always written with comma separators



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-28078: TestTxnDbUtil should generate csv files when we query the metastore database [hive]

Posted by "InvisibleProgrammer (via GitHub)" <gi...@apache.org>.
InvisibleProgrammer commented on code in PR #5085:
URL: https://github.com/apache/hive/pull/5085#discussion_r1489469862


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestTxnDbUtil.java:
##########
@@ -370,6 +384,8 @@ public static String queryToString(Configuration conf, String query, boolean inc
     return sb.toString();
   }
 
+

Review Comment:
   Thank you for spotting. It was a mistake. I kept one line as a spacer between this and the next method. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-28078: TestTxnDbUtil should generate csv files when we query the metastore database [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5085:
URL: https://github.com/apache/hive/pull/5085#issuecomment-1944280507

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5085) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5085&resolved=false&inNewCodePeriod=true)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5085&resolved=false&inNewCodePeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5085)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org