Re: Review Request 34457: Print the query execution status on CLI

[Raju Bairishetti <ra...@gmail.com>]

> On May 20, 2015, 5:23 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
> 
> Amareshwari Sriramadasu wrote:
>     I meant while opening 
>     $lens-cli  // can open in verbose loggind mode i.e. logging on console
>     $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
> 
> Rajat Khandelwal wrote:
>     +1.
> 
> Himanshu Gahlaut wrote:
>     would completely agree to the fact that passing verbose flag to functions is a less optimal approach. Will try to post a suggestion soon.
> 
> Himanshu Gahlaut wrote:
>     Another approach could be to change the logging level of cliLogger programmatically to DEBUG from INFO. This can be done at only one place in BaseLensCommand#beforeInvocation method which intercepts all commands. We can log everything we want to display at DEBUG Level and keep ConsoleAppender's Threshold at INFO level in file. The file appender can have its own Threshold for logging as per its need.
> 
> Raju Bairishetti wrote:
>     Yes, passing verbose to all the fuctions is not a good idea. I have done these changes so that log information for other ocmmands won't be display on the cli.
>     
>     We can enable the verbose by configuring the log4j properties. Right now, it is not displaying any logs on the cli. In that case, we do not need even clilogger for logging. We can directly use class level loggers for logging. Otherwise we can simply configure one more cli logger and need to change the existing logs for all commands to use cli logger. This amy affect server side log information like we do not know the class level log information(i.e. everything shows clilogger info). 
>     
>     @Himanshu any suggestions?
> 
> Himanshu Gahlaut wrote:
>     @Raju, Need your help in defining the goal of verbose option in command clearly. Is the goal debugging or is it to provide intermediate information which might be used to get more information similar to a joub url when we run pig scripts ? 
>     
>     Explaining below, how a clear goal will help.
>     
>     (1) If the goal is debugging, then as you already suggested above, class level loggers and programmatically changing the log level to debug for ConsoleAppender would be good enough. All loggers will start logging on console which will help in debugging. 
>     
>     (2) If the goal is intermediate informational output, then a separate logger like a cliLogger would help. Any code which wants to print some informational output can use this logger and we can programmatically change its level to DEBUG in command interceptor when verbose option is ON.
>     
>     More inclined towards (1), however would prefer to hear more from you.
> 
> Rajat Khandelwal wrote:
>     I think we should have a debug option for cli, with which debugging can be done. Other than that, query progress should not be part of debug session. query progress should be shown if `--progress` is true in the command.
> 
> Raju Bairishetti wrote:
>     I think initial goal is to give some information about the query status to the users. User can get the logs on cli by having appeneders in log4j properties. Seems that does not require any code change.
> 
> Rajat Khandelwal wrote:
>     The way I see it, is that there can be an interface `QueryProgress` in lens-server-api and `LensDriver` can have a method returning `QueryProgress` The only thing `QueryProgress` *needs* to implement is `float/double getProgress()`. Each driver can have its own implementation, For example, doing a query status on hive query returns three rows, one of which is progress. The driver can parse and return that progress. Similar thing can be there for JDBC. CLI just needs to ask for `QueryProgress` object from server which in turn will ask it from the driver. Driver can give it's own implementation, which will eventually be returned by server to cli.

Yes, Right now Query status contains progress information as well. @Rajat Can we use the same for knowing the query progress information(i.e. QueryStatus.getProgessMessage()) or do we need to introduce a separate interface?


- Raju


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------


On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
> 
> (Updated May 20, 2015, 4:48 a.m.)
> 
> 
> Review request for lens and Amareshwari Sriramadasu.
> 
> 
> Bugs: LENS-25
>     https://issues.apache.org/jira/browse/LENS-25
> 
> 
> Repository: lens
> 
> 
> Description
> -------
> 
> Added a cli logger for showing the query execution status
> Added a verbose option
> 
> 
> Diffs
> -----
> 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c 
>   lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a 
>   lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab 
>   lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82 
>   tools/conf/client/log4j.properties 5520304 
> 
> Diff: https://reviews.apache.org/r/34457/diff/
> 
> 
> Testing
> -------
> 
> Tested on cli
> 
> **Enable logging by settign verbose value to true (by default to true)**
> 
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [      LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [      LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [      LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category    _c1    
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
> 
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name    sample_dim2.name    
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
> 
> 
> Thanks,
> 
> Raju Bairishetti
> 
>