You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Shreepadma Venugopalan <sh...@cloudera.com> on 2013/09/25 02:08:23 UTC

Review Request 14326: HIVE-4629. HS2 should support an API to retrieve query logs

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14326/
-----------------------------------------------------------

Review request for hive and Brock Noland.


Bugs: HIVE-4629
    https://issues.apache.org/jira/browse/HIVE-4629


Repository: hive-git


Description
-------

Adds a new API to HS2, String getLog(OperationHandle opHandle) that returns the query log for a given operation handle. The log is maintained in memory as a circular buffer. The default size is 128 KB, but can be configured by the user. Logging is initialized if hive.server2.in.mem.logging is set to true.

Log object is created in executeStatement,getColumns,getTables,getSchemas,getCatalogs,getTypeInfo,getFunctions and destroyed in closeOperation, cancelOperation.


Diffs
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e971644 
  conf/hive-default.xml.template 1ee756c 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2912ece 
  jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java 09ab3c2 
  service/if/TCLIService.thrift 6e20375 
  service/src/java/org/apache/hive/service/cli/CLIService.java 1a7f338 
  service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 14ef54f 
  service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 9dca874 
  service/src/java/org/apache/hive/service/cli/ICLIService.java f647ce6 
  service/src/java/org/apache/hive/service/cli/log/LinkedStringBuffer.java PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/log/LogDivertAppender.java PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/log/LogManager.java PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/log/OperationLog.java PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 1f78a1d 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 00058cc 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 11c96b2 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java f392d62 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 2f2866f 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 9bb2a0f 

Diff: https://reviews.apache.org/r/14326/diff/


Testing
-------

Add a new unit test to test log retrieval.


Thanks,

Shreepadma Venugopalan


Re: Review Request 14326: HIVE-4629. HS2 should support an API to retrieve query logs

Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14326/#review44414
-----------------------------------------------------------



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
<https://reviews.apache.org/r/14326/#comment78792>

    It will be also useful to have the size unit indication in the name ( eg log.size.kb)
    



jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java
<https://reviews.apache.org/r/14326/#comment78804>

    So far this has been considered an hive internal class (not an external user api). This change seems to be aiming at making this function available to non-hive users.
    I think that is OK, but then we should document that explicitly.
    



service/src/java/org/apache/hive/service/cli/CLIService.java
<https://reviews.apache.org/r/14326/#comment78817>

    I don't think the logs from commands such as getOperationStatus should go into this operation log.
    I don't think it will be useful for use cases such as Hue.
    



service/src/java/org/apache/hive/service/cli/log/LogManager.java
<https://reviews.apache.org/r/14326/#comment78816>

    I agree with Carl, I think this is better as an object in OperationManager rather than a service.
    



service/src/java/org/apache/hive/service/cli/log/LogManager.java
<https://reviews.apache.org/r/14326/#comment78815>

    Isn't this re-implementing the functionality of ThreadLocal ? Should we just use ThreadLocal for it ? Does that not work because log4j does the logging asynchronously ?
    


- Thejas Nair


On Sept. 25, 2013, 12:08 a.m., Shreepadma Venugopalan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14326/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2013, 12:08 a.m.)
> 
> 
> Review request for hive and Brock Noland.
> 
> 
> Bugs: HIVE-4629
>     https://issues.apache.org/jira/browse/HIVE-4629
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Adds a new API to HS2, String getLog(OperationHandle opHandle) that returns the query log for a given operation handle. The log is maintained in memory as a circular buffer. The default size is 128 KB, but can be configured by the user. Logging is initialized if hive.server2.in.mem.logging is set to true.
> 
> Log object is created in executeStatement,getColumns,getTables,getSchemas,getCatalogs,getTypeInfo,getFunctions and destroyed in closeOperation, cancelOperation.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e971644 
>   conf/hive-default.xml.template 1ee756c 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2912ece 
>   jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java 09ab3c2 
>   service/if/TCLIService.thrift 6e20375 
>   service/src/java/org/apache/hive/service/cli/CLIService.java 1a7f338 
>   service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 14ef54f 
>   service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 9dca874 
>   service/src/java/org/apache/hive/service/cli/ICLIService.java f647ce6 
>   service/src/java/org/apache/hive/service/cli/log/LinkedStringBuffer.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/log/LogDivertAppender.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/log/LogManager.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/log/OperationLog.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 1f78a1d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 00058cc 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 11c96b2 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java f392d62 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 2f2866f 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 9bb2a0f 
> 
> Diff: https://reviews.apache.org/r/14326/diff/
> 
> 
> Testing
> -------
> 
> Add a new unit test to test log retrieval.
> 
> 
> Thanks,
> 
> Shreepadma Venugopalan
> 
>


Re: Review Request 14326: HIVE-4629. HS2 should support an API to retrieve query logs

Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14326/#review26683
-----------------------------------------------------------


Shreepadma,

This looks like a very useful feature for clients of HS2! Indeed I wish all databases had this feature. The patch looks very good. I have noted a few items below, nothing major, basically a bunch of nits.

Brock


jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java
<https://reviews.apache.org/r/14326/#comment51991>

    Let's add a test where we call getLog without any query.



jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java
<https://reviews.apache.org/r/14326/#comment51987>

    Why even catch this if we rethrow immediately.



jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java
<https://reviews.apache.org/r/14326/#comment51988>

    I think we should append these two error messages with log.



service/if/TCLIService.thrift
<https://reviews.apache.org/r/14326/#comment51976>

    Let's trim this trailing ws. (understand it's not yours.)



service/src/java/org/apache/hive/service/cli/CLIService.java
<https://reviews.apache.org/r/14326/#comment51977>

    Should this be called in a finally block? Same question for the items below.



service/src/java/org/apache/hive/service/cli/CLIService.java
<https://reviews.apache.org/r/14326/#comment51986>

    Is this useful at the INFO level? It's completely your call but I was just curious.



service/src/java/org/apache/hive/service/cli/log/LinkedStringBuffer.java
<https://reviews.apache.org/r/14326/#comment51984>

    Looks like we only require the List interface on the LHS?



service/src/java/org/apache/hive/service/cli/log/LinkedStringBuffer.java
<https://reviews.apache.org/r/14326/#comment51985>

    Internally we storing this as an int. Should the return type be a int or the internal type be a long?



service/src/java/org/apache/hive/service/cli/log/LogDivertAppender.java
<https://reviews.apache.org/r/14326/#comment51973>

    When we drop an event, that means it's not stored in memory. Is it still logged to the HS2 log file?
    
    Since we have a limit on the amount of data we store per log capture, I guess the question is, is all this data also sent to the HS2 log file?



service/src/java/org/apache/hive/service/cli/log/LogManager.java
<https://reviews.apache.org/r/14326/#comment51983>

    The order of members for this class should be:
    
    static final
    final
    non-final



service/src/java/org/apache/hive/service/cli/log/LogManager.java
<https://reviews.apache.org/r/14326/#comment51981>

    These two should start with a lower case cahr



service/src/java/org/apache/hive/service/cli/log/LogManager.java
<https://reviews.apache.org/r/14326/#comment51982>

    LOG should be final



service/src/java/org/apache/hive/service/cli/log/LogManager.java
<https://reviews.apache.org/r/14326/#comment51980>

    The variable should start with a lowercase char. Same same below and above.



service/src/java/org/apache/hive/service/cli/log/LogManager.java
<https://reviews.apache.org/r/14326/#comment51969>

    spelling



service/src/java/org/apache/hive/service/cli/log/LogManager.java
<https://reviews.apache.org/r/14326/#comment51989>

    Slightly confusing. I would expect:
    
    if(operationLog == nul) {
      if(createIfAbsent) {
        doCreate();
      } else {
        throw
      }
    }



service/src/java/org/apache/hive/service/cli/log/LogManager.java
<https://reviews.apache.org/r/14326/#comment51990>

    Operation is spelled wrong



service/src/java/org/apache/hive/service/cli/session/HiveSession.java
<https://reviews.apache.org/r/14326/#comment51970>

    trailing ws



service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
<https://reviews.apache.org/r/14326/#comment51979>

    Similar to above, should this be part of the finally?



service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
<https://reviews.apache.org/r/14326/#comment51971>

    I know we are printing stack traces to syserr elsewhere in this file but lets' use the logger?



service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
<https://reviews.apache.org/r/14326/#comment51972>

    What encoding should this be, UTF-8?


- Brock Noland


On Sept. 25, 2013, 12:08 a.m., Shreepadma Venugopalan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14326/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2013, 12:08 a.m.)
> 
> 
> Review request for hive and Brock Noland.
> 
> 
> Bugs: HIVE-4629
>     https://issues.apache.org/jira/browse/HIVE-4629
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Adds a new API to HS2, String getLog(OperationHandle opHandle) that returns the query log for a given operation handle. The log is maintained in memory as a circular buffer. The default size is 128 KB, but can be configured by the user. Logging is initialized if hive.server2.in.mem.logging is set to true.
> 
> Log object is created in executeStatement,getColumns,getTables,getSchemas,getCatalogs,getTypeInfo,getFunctions and destroyed in closeOperation, cancelOperation.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e971644 
>   conf/hive-default.xml.template 1ee756c 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2912ece 
>   jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java 09ab3c2 
>   service/if/TCLIService.thrift 6e20375 
>   service/src/java/org/apache/hive/service/cli/CLIService.java 1a7f338 
>   service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 14ef54f 
>   service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 9dca874 
>   service/src/java/org/apache/hive/service/cli/ICLIService.java f647ce6 
>   service/src/java/org/apache/hive/service/cli/log/LinkedStringBuffer.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/log/LogDivertAppender.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/log/LogManager.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/log/OperationLog.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 1f78a1d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 00058cc 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 11c96b2 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java f392d62 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 2f2866f 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 9bb2a0f 
> 
> Diff: https://reviews.apache.org/r/14326/diff/
> 
> 
> Testing
> -------
> 
> Add a new unit test to test log retrieval.
> 
> 
> Thanks,
> 
> Shreepadma Venugopalan
> 
>


Re: Review Request 14326: HIVE-4629. HS2 should support an API to retrieve query logs

Posted by Carl Steinbach <ca...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14326/#review27210
-----------------------------------------------------------



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
<https://reviews.apache.org/r/14326/#comment52909>

    Is this in memory logging for the server process, sessions, operations, or ...? Please make the property name more specific, and please change the ordering of the subnames so that they run from general to specific, e.g. hive.server2.logging.operation.*



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
<https://reviews.apache.org/r/14326/#comment52910>

    Please use kilobytes instead of bytes.



conf/hive-default.xml.template
<https://reviews.apache.org/r/14326/#comment52911>

    It's an operation log, not a query log. Also, what are the units of this value?



jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java
<https://reviews.apache.org/r/14326/#comment52928>

    This feature needs to be tested at the Thrift level. Please include a test where getOperationLog is called more than once for the same operation, and a test where getOperationLog is demonstrated with asynchronous queries.



jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java
<https://reviews.apache.org/r/14326/#comment52927>

    This looks like a substring from the previous check.



service/if/TCLIService.thrift
<https://reviews.apache.org/r/14326/#comment52912>

    Please change the name to GetOperationLog()



service/if/TCLIService.thrift
<https://reviews.apache.org/r/14326/#comment52914>

    Does this return one line from the log per call, or do I get the whole 128kb buffer? What happens if I call this RPC multiple times on the same operation handle? What happens if the circular buffer rolls over between two calls?



service/if/TCLIService.thrift
<https://reviews.apache.org/r/14326/#comment52913>

    Formatting



service/src/java/org/apache/hive/service/cli/CLIService.java
<https://reviews.apache.org/r/14326/#comment52915>

    Why isn't OperationLog a member of the o.a.h.service.cli.operation package?



service/src/java/org/apache/hive/service/cli/log/LinkedStringBuffer.java
<https://reviews.apache.org/r/14326/#comment52917>

    The name of this class should describe the functionality as opposed to the implementation.



service/src/java/org/apache/hive/service/cli/log/LinkedStringBuffer.java
<https://reviews.apache.org/r/14326/#comment52916>

    StringBuilder provides a delete(start, end) method which can be used to easily implement a ring buffer. I think that would be a better option here since it would eliminate the need to iterate over every log entry in read().



service/src/java/org/apache/hive/service/cli/log/LogManager.java
<https://reviews.apache.org/r/14326/#comment52919>

    I think most of this code belongs in OperationManager as opposed to its own class.



service/src/java/org/apache/hive/service/cli/log/LogManager.java
<https://reviews.apache.org/r/14326/#comment52918>

    Thread names are mutable and don't have to be unique. I think you want to use threadId instead.



service/src/java/org/apache/hive/service/cli/log/LogManager.java
<https://reviews.apache.org/r/14326/#comment52920>

    spelling



service/src/java/org/apache/hive/service/cli/log/OperationLog.java
<https://reviews.apache.org/r/14326/#comment52921>

    Please move this to o.a.h.service.cli.operation.*



service/src/java/org/apache/hive/service/cli/log/OperationLog.java
<https://reviews.apache.org/r/14326/#comment52922>

    Does this comment imply that when operation logging is enabled log messages generated by  operations won't appear in the system logs?
    
    Also, can you please include a paragraph somewhere that provides a high-level overview of the operation logging mechanism? What expectations does it make about the relationship between Sessions/Operations and threads, and does it support asynchronous execution?



service/src/java/org/apache/hive/service/cli/operation/OperationManager.java
<https://reviews.apache.org/r/14326/#comment52923>

    Seems like most of the operation logging code should go in this class.



service/src/java/org/apache/hive/service/cli/session/HiveSession.java
<https://reviews.apache.org/r/14326/#comment52924>

    I don't understand why the OperationLogger hangs off the SessionManager instead of the OperationManager.



service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
<https://reviews.apache.org/r/14326/#comment52925>

    If the established protocol for registering a thread is to always unregister it first, why not have registerCurrentThread call unregisterCurrentThread?



service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
<https://reviews.apache.org/r/14326/#comment52926>

    Does operation logging work for executeStatementAsync()?


- Carl Steinbach


On Sept. 25, 2013, 12:08 a.m., Shreepadma Venugopalan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14326/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2013, 12:08 a.m.)
> 
> 
> Review request for hive and Brock Noland.
> 
> 
> Bugs: HIVE-4629
>     https://issues.apache.org/jira/browse/HIVE-4629
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Adds a new API to HS2, String getLog(OperationHandle opHandle) that returns the query log for a given operation handle. The log is maintained in memory as a circular buffer. The default size is 128 KB, but can be configured by the user. Logging is initialized if hive.server2.in.mem.logging is set to true.
> 
> Log object is created in executeStatement,getColumns,getTables,getSchemas,getCatalogs,getTypeInfo,getFunctions and destroyed in closeOperation, cancelOperation.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e971644 
>   conf/hive-default.xml.template 1ee756c 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2912ece 
>   jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java 09ab3c2 
>   service/if/TCLIService.thrift 6e20375 
>   service/src/java/org/apache/hive/service/cli/CLIService.java 1a7f338 
>   service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 14ef54f 
>   service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 9dca874 
>   service/src/java/org/apache/hive/service/cli/ICLIService.java f647ce6 
>   service/src/java/org/apache/hive/service/cli/log/LinkedStringBuffer.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/log/LogDivertAppender.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/log/LogManager.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/log/OperationLog.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 1f78a1d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 00058cc 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 11c96b2 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java f392d62 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 2f2866f 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 9bb2a0f 
> 
> Diff: https://reviews.apache.org/r/14326/diff/
> 
> 
> Testing
> -------
> 
> Add a new unit test to test log retrieval.
> 
> 
> Thanks,
> 
> Shreepadma Venugopalan
> 
>