You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Raju Bairishetti <ra...@gmail.com> on 2015/05/20 06:48:21 UTC
Review Request 34457: Print the query execution status on CLI
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/
-----------------------------------------------------------
Review request for lens and Amareshwari Sriramadasu.
Bugs: LENS-25
https://issues.apache.org/jira/browse/LENS-25
Repository: lens
Description
-------
Added a cli logger for showing the query execution status
Added a verbose option
Diffs
-----
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
tools/conf/client/log4j.properties 5520304
Diff: https://reviews.apache.org/r/34457/diff/
Testing
-------
Tested on cli
**Enable logging by settign verbose value to true (by default to true)**
**lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
product_details.category _c1
Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
**Disable logging by setting verbose value to false (by default to true)**
lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
sample_dim.name sample_dim2.name
Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
Thanks,
Raju Bairishetti
Re: Review Request 34457: Print the query execution status on CLI
Posted by Raju Bairishetti <ra...@gmail.com>.
> On May 20, 2015, 10:58 a.m., Rajat Khandelwal wrote:
> > lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java, line 52
> > <https://reviews.apache.org/r/34457/diff/1/?file=964884#file964884line52>
> >
> > Seems you haven't pulled from latest. Please do so. There are a lot of changes that might conflict with your changes.
Sure, will rebase it to master.
- Raju
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84490
-----------------------------------------------------------
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84490
-----------------------------------------------------------
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java
<https://reviews.apache.org/r/34457/#comment135806>
Seems you haven't pulled from latest. Please do so. There are a lot of changes that might conflict with your changes.
- Rajat Khandelwal
On May 20, 2015, 10:18 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 10:18 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Himanshu Gahlaut <ga...@gmail.com>.
> On May 20, 2015, 5:23 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
>
> Amareshwari Sriramadasu wrote:
> I meant while opening
> $lens-cli // can open in verbose loggind mode i.e. logging on console
> $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
>
> Rajat Khandelwal wrote:
> +1.
>
> Himanshu Gahlaut wrote:
> would completely agree to the fact that passing verbose flag to functions is a less optimal approach. Will try to post a suggestion soon.
Another approach could be to change the logging level of cliLogger programmatically to DEBUG from INFO. This can be done at only one place in BaseLensCommand#beforeInvocation method which intercepts all commands. We can log everything we want to display at DEBUG Level and keep ConsoleAppender's Threshold at INFO level in file. The file appender can have its own Threshold for logging as per its need.
- Himanshu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Raju Bairishetti <ra...@gmail.com>.
> On May 20, 2015, 5:23 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
>
> Amareshwari Sriramadasu wrote:
> I meant while opening
> $lens-cli // can open in verbose loggind mode i.e. logging on console
> $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
>
> Rajat Khandelwal wrote:
> +1.
>
> Himanshu Gahlaut wrote:
> would completely agree to the fact that passing verbose flag to functions is a less optimal approach. Will try to post a suggestion soon.
>
> Himanshu Gahlaut wrote:
> Another approach could be to change the logging level of cliLogger programmatically to DEBUG from INFO. This can be done at only one place in BaseLensCommand#beforeInvocation method which intercepts all commands. We can log everything we want to display at DEBUG Level and keep ConsoleAppender's Threshold at INFO level in file. The file appender can have its own Threshold for logging as per its need.
>
> Raju Bairishetti wrote:
> Yes, passing verbose to all the fuctions is not a good idea. I have done these changes so that log information for other ocmmands won't be display on the cli.
>
> We can enable the verbose by configuring the log4j properties. Right now, it is not displaying any logs on the cli. In that case, we do not need even clilogger for logging. We can directly use class level loggers for logging. Otherwise we can simply configure one more cli logger and need to change the existing logs for all commands to use cli logger. This amy affect server side log information like we do not know the class level log information(i.e. everything shows clilogger info).
>
> @Himanshu any suggestions?
>
> Himanshu Gahlaut wrote:
> @Raju, Need your help in defining the goal of verbose option in command clearly. Is the goal debugging or is it to provide intermediate information which might be used to get more information similar to a joub url when we run pig scripts ?
>
> Explaining below, how a clear goal will help.
>
> (1) If the goal is debugging, then as you already suggested above, class level loggers and programmatically changing the log level to debug for ConsoleAppender would be good enough. All loggers will start logging on console which will help in debugging.
>
> (2) If the goal is intermediate informational output, then a separate logger like a cliLogger would help. Any code which wants to print some informational output can use this logger and we can programmatically change its level to DEBUG in command interceptor when verbose option is ON.
>
> More inclined towards (1), however would prefer to hear more from you.
>
> Rajat Khandelwal wrote:
> I think we should have a debug option for cli, with which debugging can be done. Other than that, query progress should not be part of debug session. query progress should be shown if `--progress` is true in the command.
>
> Raju Bairishetti wrote:
> I think initial goal is to give some information about the query status to the users. User can get the logs on cli by having appeneders in log4j properties. Seems that does not require any code change.
>
> Rajat Khandelwal wrote:
> The way I see it, is that there can be an interface `QueryProgress` in lens-server-api and `LensDriver` can have a method returning `QueryProgress` The only thing `QueryProgress` *needs* to implement is `float/double getProgress()`. Each driver can have its own implementation, For example, doing a query status on hive query returns three rows, one of which is progress. The driver can parse and return that progress. Similar thing can be there for JDBC. CLI just needs to ask for `QueryProgress` object from server which in turn will ask it from the driver. Driver can give it's own implementation, which will eventually be returned by server to cli.
Yes, Right now Query status contains progress information as well. @Rajat Can we use the same for knowing the query progress information(i.e. QueryStatus.getProgessMessage()) or do we need to introduce a separate interface?
- Raju
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On May 20, 2015, 5:23 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
I meant while opening
$lens-cli // can open in verbose loggind mode i.e. logging on console
$lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Himanshu Gahlaut <ga...@gmail.com>.
> On May 20, 2015, 5:23 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
>
> Amareshwari Sriramadasu wrote:
> I meant while opening
> $lens-cli // can open in verbose loggind mode i.e. logging on console
> $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
>
> Rajat Khandelwal wrote:
> +1.
>
> Himanshu Gahlaut wrote:
> would completely agree to the fact that passing verbose flag to functions is a less optimal approach. Will try to post a suggestion soon.
>
> Himanshu Gahlaut wrote:
> Another approach could be to change the logging level of cliLogger programmatically to DEBUG from INFO. This can be done at only one place in BaseLensCommand#beforeInvocation method which intercepts all commands. We can log everything we want to display at DEBUG Level and keep ConsoleAppender's Threshold at INFO level in file. The file appender can have its own Threshold for logging as per its need.
>
> Raju Bairishetti wrote:
> Yes, passing verbose to all the fuctions is not a good idea. I have done these changes so that log information for other ocmmands won't be display on the cli.
>
> We can enable the verbose by configuring the log4j properties. Right now, it is not displaying any logs on the cli. In that case, we do not need even clilogger for logging. We can directly use class level loggers for logging. Otherwise we can simply configure one more cli logger and need to change the existing logs for all commands to use cli logger. This amy affect server side log information like we do not know the class level log information(i.e. everything shows clilogger info).
>
> @Himanshu any suggestions?
@Raju, Need your help in defining the goal of verbose option in command clearly. Is the goal debugging or is it to provide intermediate information which might be used to get more information similar to a joub url when we run pig scripts ?
Explaining below, how a clear goal will help.
(1) If the goal is debugging, then as you already suggested above, class level loggers and programmatically changing the log level to debug for ConsoleAppender would be good enough. All loggers will start logging on console which will help in debugging.
(2) If the goal is intermediate informational output, then a separate logger like a cliLogger would help. Any code which wants to print some informational output can use this logger and we can programmatically change its level to DEBUG in command interceptor when verbose option is ON.
More inclined towards (1), however would prefer to hear more from you.
- Himanshu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Rajat Khandelwal <ra...@gmail.com>.
> On May 20, 2015, 10:53 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
>
> Amareshwari Sriramadasu wrote:
> I meant while opening
> $lens-cli // can open in verbose loggind mode i.e. logging on console
> $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
>
> Rajat Khandelwal wrote:
> +1.
>
> Himanshu Gahlaut wrote:
> would completely agree to the fact that passing verbose flag to functions is a less optimal approach. Will try to post a suggestion soon.
>
> Himanshu Gahlaut wrote:
> Another approach could be to change the logging level of cliLogger programmatically to DEBUG from INFO. This can be done at only one place in BaseLensCommand#beforeInvocation method which intercepts all commands. We can log everything we want to display at DEBUG Level and keep ConsoleAppender's Threshold at INFO level in file. The file appender can have its own Threshold for logging as per its need.
>
> Raju Bairishetti wrote:
> Yes, passing verbose to all the fuctions is not a good idea. I have done these changes so that log information for other ocmmands won't be display on the cli.
>
> We can enable the verbose by configuring the log4j properties. Right now, it is not displaying any logs on the cli. In that case, we do not need even clilogger for logging. We can directly use class level loggers for logging. Otherwise we can simply configure one more cli logger and need to change the existing logs for all commands to use cli logger. This amy affect server side log information like we do not know the class level log information(i.e. everything shows clilogger info).
>
> @Himanshu any suggestions?
>
> Himanshu Gahlaut wrote:
> @Raju, Need your help in defining the goal of verbose option in command clearly. Is the goal debugging or is it to provide intermediate information which might be used to get more information similar to a joub url when we run pig scripts ?
>
> Explaining below, how a clear goal will help.
>
> (1) If the goal is debugging, then as you already suggested above, class level loggers and programmatically changing the log level to debug for ConsoleAppender would be good enough. All loggers will start logging on console which will help in debugging.
>
> (2) If the goal is intermediate informational output, then a separate logger like a cliLogger would help. Any code which wants to print some informational output can use this logger and we can programmatically change its level to DEBUG in command interceptor when verbose option is ON.
>
> More inclined towards (1), however would prefer to hear more from you.
>
> Rajat Khandelwal wrote:
> I think we should have a debug option for cli, with which debugging can be done. Other than that, query progress should not be part of debug session. query progress should be shown if `--progress` is true in the command.
>
> Raju Bairishetti wrote:
> I think initial goal is to give some information about the query status to the users. User can get the logs on cli by having appeneders in log4j properties. Seems that does not require any code change.
The way I see it, is that there can be an interface `QueryProgress` in lens-server-api and `LensDriver` can have a method returning `QueryProgress` The only thing `QueryProgress` *needs* to implement is `float/double getProgress()`. Each driver can have its own implementation, For example, doing a query status on hive query returns three rows, one of which is progress. The driver can parse and return that progress. Similar thing can be there for JDBC. CLI just needs to ask for `QueryProgress` object from server which in turn will ask it from the driver. Driver can give it's own implementation, which will eventually be returned by server to cli.
- Rajat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 10:18 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 10:18 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Rajat Khandelwal <ra...@gmail.com>.
> On May 20, 2015, 10:53 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
>
> Amareshwari Sriramadasu wrote:
> I meant while opening
> $lens-cli // can open in verbose loggind mode i.e. logging on console
> $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
>
> Rajat Khandelwal wrote:
> +1.
>
> Himanshu Gahlaut wrote:
> would completely agree to the fact that passing verbose flag to functions is a less optimal approach. Will try to post a suggestion soon.
>
> Himanshu Gahlaut wrote:
> Another approach could be to change the logging level of cliLogger programmatically to DEBUG from INFO. This can be done at only one place in BaseLensCommand#beforeInvocation method which intercepts all commands. We can log everything we want to display at DEBUG Level and keep ConsoleAppender's Threshold at INFO level in file. The file appender can have its own Threshold for logging as per its need.
>
> Raju Bairishetti wrote:
> Yes, passing verbose to all the fuctions is not a good idea. I have done these changes so that log information for other ocmmands won't be display on the cli.
>
> We can enable the verbose by configuring the log4j properties. Right now, it is not displaying any logs on the cli. In that case, we do not need even clilogger for logging. We can directly use class level loggers for logging. Otherwise we can simply configure one more cli logger and need to change the existing logs for all commands to use cli logger. This amy affect server side log information like we do not know the class level log information(i.e. everything shows clilogger info).
>
> @Himanshu any suggestions?
>
> Himanshu Gahlaut wrote:
> @Raju, Need your help in defining the goal of verbose option in command clearly. Is the goal debugging or is it to provide intermediate information which might be used to get more information similar to a joub url when we run pig scripts ?
>
> Explaining below, how a clear goal will help.
>
> (1) If the goal is debugging, then as you already suggested above, class level loggers and programmatically changing the log level to debug for ConsoleAppender would be good enough. All loggers will start logging on console which will help in debugging.
>
> (2) If the goal is intermediate informational output, then a separate logger like a cliLogger would help. Any code which wants to print some informational output can use this logger and we can programmatically change its level to DEBUG in command interceptor when verbose option is ON.
>
> More inclined towards (1), however would prefer to hear more from you.
I think we should have a debug option for cli, with which debugging can be done. Other than that, query progress should not be part of debug session. query progress should be shown if `--progress` is true in the command.
- Rajat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 10:18 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 10:18 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On May 20, 2015, 5:23 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
>
> Amareshwari Sriramadasu wrote:
> I meant while opening
> $lens-cli // can open in verbose loggind mode i.e. logging on console
> $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
>
> Rajat Khandelwal wrote:
> +1.
>
> Himanshu Gahlaut wrote:
> would completely agree to the fact that passing verbose flag to functions is a less optimal approach. Will try to post a suggestion soon.
>
> Himanshu Gahlaut wrote:
> Another approach could be to change the logging level of cliLogger programmatically to DEBUG from INFO. This can be done at only one place in BaseLensCommand#beforeInvocation method which intercepts all commands. We can log everything we want to display at DEBUG Level and keep ConsoleAppender's Threshold at INFO level in file. The file appender can have its own Threshold for logging as per its need.
>
> Raju Bairishetti wrote:
> Yes, passing verbose to all the fuctions is not a good idea. I have done these changes so that log information for other ocmmands won't be display on the cli.
>
> We can enable the verbose by configuring the log4j properties. Right now, it is not displaying any logs on the cli. In that case, we do not need even clilogger for logging. We can directly use class level loggers for logging. Otherwise we can simply configure one more cli logger and need to change the existing logs for all commands to use cli logger. This amy affect server side log information like we do not know the class level log information(i.e. everything shows clilogger info).
>
> @Himanshu any suggestions?
>
> Himanshu Gahlaut wrote:
> @Raju, Need your help in defining the goal of verbose option in command clearly. Is the goal debugging or is it to provide intermediate information which might be used to get more information similar to a joub url when we run pig scripts ?
>
> Explaining below, how a clear goal will help.
>
> (1) If the goal is debugging, then as you already suggested above, class level loggers and programmatically changing the log level to debug for ConsoleAppender would be good enough. All loggers will start logging on console which will help in debugging.
>
> (2) If the goal is intermediate informational output, then a separate logger like a cliLogger would help. Any code which wants to print some informational output can use this logger and we can programmatically change its level to DEBUG in command interceptor when verbose option is ON.
>
> More inclined towards (1), however would prefer to hear more from you.
>
> Rajat Khandelwal wrote:
> I think we should have a debug option for cli, with which debugging can be done. Other than that, query progress should not be part of debug session. query progress should be shown if `--progress` is true in the command.
>
> Raju Bairishetti wrote:
> I think initial goal is to give some information about the query status to the users. User can get the logs on cli by having appeneders in log4j properties. Seems that does not require any code change.
>
> Rajat Khandelwal wrote:
> The way I see it, is that there can be an interface `QueryProgress` in lens-server-api and `LensDriver` can have a method returning `QueryProgress` The only thing `QueryProgress` *needs* to implement is `float/double getProgress()`. Each driver can have its own implementation, For example, doing a query status on hive query returns three rows, one of which is progress. The driver can parse and return that progress. Similar thing can be there for JDBC. CLI just needs to ask for `QueryProgress` object from server which in turn will ask it from the driver. Driver can give it's own implementation, which will eventually be returned by server to cli.
>
> Raju Bairishetti wrote:
> Yes, Right now Query status contains progress information as well. @Rajat Can we use the same for knowing the query progress information(i.e. QueryStatus.getProgessMessage()) or do we need to introduce a separate interface?
>
> Rajat Khandelwal wrote:
> Let's use what we already have :). But in cli's non verbose mode let's show the progress on cli in case of sync queries(http://stackoverflow.com/questions/852665/command-line-progress-bar-in-java). For async queries, `query progress` command can be there for the same purpose. `query status` prints the whole status.
>
> In verbose/debug mode, sync queries are showing the whole status, rest all remains the same.
Summarizing concerns and discussion above.
We would like to have --verbose option to know more about whats happening a command, and --debug option to actually look at debug logs. Here is how it can be achiveed :
- All client and CLI classes continues to use class level loggers for info, debug logging for debugging purpose. --debug option can add console appender to see the logs directly on console. If no option is available (as it is now), use can add console appendar in log4j, all the logs will start showing up now. But can be enhanced to add --debug option which sets the console appender.
- For showing more information about a command, we need a separate logger than class level logger. --verbose option can add console appender for this separate verbose logger. Commands will be using this verbose logger to give intermediate information about the command.
we are trying to add verbose option and verbose logger in this jira. If adding debug option has to be seperate jira, I'm fine with it.
Does it make sense?
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Raju Bairishetti <ra...@gmail.com>.
> On May 20, 2015, 5:23 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
>
> Amareshwari Sriramadasu wrote:
> I meant while opening
> $lens-cli // can open in verbose loggind mode i.e. logging on console
> $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
>
> Rajat Khandelwal wrote:
> +1.
>
> Himanshu Gahlaut wrote:
> would completely agree to the fact that passing verbose flag to functions is a less optimal approach. Will try to post a suggestion soon.
>
> Himanshu Gahlaut wrote:
> Another approach could be to change the logging level of cliLogger programmatically to DEBUG from INFO. This can be done at only one place in BaseLensCommand#beforeInvocation method which intercepts all commands. We can log everything we want to display at DEBUG Level and keep ConsoleAppender's Threshold at INFO level in file. The file appender can have its own Threshold for logging as per its need.
>
> Raju Bairishetti wrote:
> Yes, passing verbose to all the fuctions is not a good idea. I have done these changes so that log information for other ocmmands won't be display on the cli.
>
> We can enable the verbose by configuring the log4j properties. Right now, it is not displaying any logs on the cli. In that case, we do not need even clilogger for logging. We can directly use class level loggers for logging. Otherwise we can simply configure one more cli logger and need to change the existing logs for all commands to use cli logger. This amy affect server side log information like we do not know the class level log information(i.e. everything shows clilogger info).
>
> @Himanshu any suggestions?
>
> Himanshu Gahlaut wrote:
> @Raju, Need your help in defining the goal of verbose option in command clearly. Is the goal debugging or is it to provide intermediate information which might be used to get more information similar to a joub url when we run pig scripts ?
>
> Explaining below, how a clear goal will help.
>
> (1) If the goal is debugging, then as you already suggested above, class level loggers and programmatically changing the log level to debug for ConsoleAppender would be good enough. All loggers will start logging on console which will help in debugging.
>
> (2) If the goal is intermediate informational output, then a separate logger like a cliLogger would help. Any code which wants to print some informational output can use this logger and we can programmatically change its level to DEBUG in command interceptor when verbose option is ON.
>
> More inclined towards (1), however would prefer to hear more from you.
>
> Rajat Khandelwal wrote:
> I think we should have a debug option for cli, with which debugging can be done. Other than that, query progress should not be part of debug session. query progress should be shown if `--progress` is true in the command.
I think initial goal is to give some information about the query status to the users. User can get the logs on cli by having appeneders in log4j properties. Seems that does not require any code change.
- Raju
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Rajat Khandelwal <ra...@gmail.com>.
> On May 20, 2015, 10:53 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
>
> Amareshwari Sriramadasu wrote:
> I meant while opening
> $lens-cli // can open in verbose loggind mode i.e. logging on console
> $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
>
> Rajat Khandelwal wrote:
> +1.
>
> Himanshu Gahlaut wrote:
> would completely agree to the fact that passing verbose flag to functions is a less optimal approach. Will try to post a suggestion soon.
>
> Himanshu Gahlaut wrote:
> Another approach could be to change the logging level of cliLogger programmatically to DEBUG from INFO. This can be done at only one place in BaseLensCommand#beforeInvocation method which intercepts all commands. We can log everything we want to display at DEBUG Level and keep ConsoleAppender's Threshold at INFO level in file. The file appender can have its own Threshold for logging as per its need.
>
> Raju Bairishetti wrote:
> Yes, passing verbose to all the fuctions is not a good idea. I have done these changes so that log information for other ocmmands won't be display on the cli.
>
> We can enable the verbose by configuring the log4j properties. Right now, it is not displaying any logs on the cli. In that case, we do not need even clilogger for logging. We can directly use class level loggers for logging. Otherwise we can simply configure one more cli logger and need to change the existing logs for all commands to use cli logger. This amy affect server side log information like we do not know the class level log information(i.e. everything shows clilogger info).
>
> @Himanshu any suggestions?
>
> Himanshu Gahlaut wrote:
> @Raju, Need your help in defining the goal of verbose option in command clearly. Is the goal debugging or is it to provide intermediate information which might be used to get more information similar to a joub url when we run pig scripts ?
>
> Explaining below, how a clear goal will help.
>
> (1) If the goal is debugging, then as you already suggested above, class level loggers and programmatically changing the log level to debug for ConsoleAppender would be good enough. All loggers will start logging on console which will help in debugging.
>
> (2) If the goal is intermediate informational output, then a separate logger like a cliLogger would help. Any code which wants to print some informational output can use this logger and we can programmatically change its level to DEBUG in command interceptor when verbose option is ON.
>
> More inclined towards (1), however would prefer to hear more from you.
>
> Rajat Khandelwal wrote:
> I think we should have a debug option for cli, with which debugging can be done. Other than that, query progress should not be part of debug session. query progress should be shown if `--progress` is true in the command.
>
> Raju Bairishetti wrote:
> I think initial goal is to give some information about the query status to the users. User can get the logs on cli by having appeneders in log4j properties. Seems that does not require any code change.
>
> Rajat Khandelwal wrote:
> The way I see it, is that there can be an interface `QueryProgress` in lens-server-api and `LensDriver` can have a method returning `QueryProgress` The only thing `QueryProgress` *needs* to implement is `float/double getProgress()`. Each driver can have its own implementation, For example, doing a query status on hive query returns three rows, one of which is progress. The driver can parse and return that progress. Similar thing can be there for JDBC. CLI just needs to ask for `QueryProgress` object from server which in turn will ask it from the driver. Driver can give it's own implementation, which will eventually be returned by server to cli.
>
> Raju Bairishetti wrote:
> Yes, Right now Query status contains progress information as well. @Rajat Can we use the same for knowing the query progress information(i.e. QueryStatus.getProgessMessage()) or do we need to introduce a separate interface?
Let's use what we already have :). But in cli's non verbose mode let's show the progress on cli in case of sync queries(http://stackoverflow.com/questions/852665/command-line-progress-bar-in-java). For async queries, `query progress` command can be there for the same purpose. `query status` prints the whole status.
In verbose/debug mode, sync queries are showing the whole status, rest all remains the same.
- Rajat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 10:18 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 10:18 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Raju Bairishetti <ra...@gmail.com>.
> On May 20, 2015, 5:23 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
>
> Amareshwari Sriramadasu wrote:
> I meant while opening
> $lens-cli // can open in verbose loggind mode i.e. logging on console
> $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
>
> Rajat Khandelwal wrote:
> +1.
>
> Himanshu Gahlaut wrote:
> would completely agree to the fact that passing verbose flag to functions is a less optimal approach. Will try to post a suggestion soon.
>
> Himanshu Gahlaut wrote:
> Another approach could be to change the logging level of cliLogger programmatically to DEBUG from INFO. This can be done at only one place in BaseLensCommand#beforeInvocation method which intercepts all commands. We can log everything we want to display at DEBUG Level and keep ConsoleAppender's Threshold at INFO level in file. The file appender can have its own Threshold for logging as per its need.
Yes, passing verbose to all the fuctions is not a good idea. I have done these changes so that log information for other ocmmands won't be display on the cli.
We can enable the verbose by configuring the log4j properties. Right now, it is not displaying any logs on the cli. In that case, we do not need even clilogger for logging. We can directly use class level loggers for logging. Otherwise we can simply configure one more cli logger and need to change the existing logs for all commands to use cli logger. This amy affect server side log information like we do not know the class level log information(i.e. everything shows clilogger info).
@Himanshu any suggestions?
- Raju
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Rajat Khandelwal <ra...@gmail.com>.
> On May 20, 2015, 10:53 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
>
> Amareshwari Sriramadasu wrote:
> I meant while opening
> $lens-cli // can open in verbose loggind mode i.e. logging on console
> $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
+1.
- Rajat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 10:18 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 10:18 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Raju Bairishetti <ra...@gmail.com>.
> On May 20, 2015, 5:23 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
>
> Amareshwari Sriramadasu wrote:
> I meant while opening
> $lens-cli // can open in verbose loggind mode i.e. logging on console
> $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
>
> Rajat Khandelwal wrote:
> +1.
>
> Himanshu Gahlaut wrote:
> would completely agree to the fact that passing verbose flag to functions is a less optimal approach. Will try to post a suggestion soon.
>
> Himanshu Gahlaut wrote:
> Another approach could be to change the logging level of cliLogger programmatically to DEBUG from INFO. This can be done at only one place in BaseLensCommand#beforeInvocation method which intercepts all commands. We can log everything we want to display at DEBUG Level and keep ConsoleAppender's Threshold at INFO level in file. The file appender can have its own Threshold for logging as per its need.
>
> Raju Bairishetti wrote:
> Yes, passing verbose to all the fuctions is not a good idea. I have done these changes so that log information for other ocmmands won't be display on the cli.
>
> We can enable the verbose by configuring the log4j properties. Right now, it is not displaying any logs on the cli. In that case, we do not need even clilogger for logging. We can directly use class level loggers for logging. Otherwise we can simply configure one more cli logger and need to change the existing logs for all commands to use cli logger. This amy affect server side log information like we do not know the class level log information(i.e. everything shows clilogger info).
>
> @Himanshu any suggestions?
>
> Himanshu Gahlaut wrote:
> @Raju, Need your help in defining the goal of verbose option in command clearly. Is the goal debugging or is it to provide intermediate information which might be used to get more information similar to a joub url when we run pig scripts ?
>
> Explaining below, how a clear goal will help.
>
> (1) If the goal is debugging, then as you already suggested above, class level loggers and programmatically changing the log level to debug for ConsoleAppender would be good enough. All loggers will start logging on console which will help in debugging.
>
> (2) If the goal is intermediate informational output, then a separate logger like a cliLogger would help. Any code which wants to print some informational output can use this logger and we can programmatically change its level to DEBUG in command interceptor when verbose option is ON.
>
> More inclined towards (1), however would prefer to hear more from you.
>
> Rajat Khandelwal wrote:
> I think we should have a debug option for cli, with which debugging can be done. Other than that, query progress should not be part of debug session. query progress should be shown if `--progress` is true in the command.
>
> Raju Bairishetti wrote:
> I think initial goal is to give some information about the query status to the users. User can get the logs on cli by having appeneders in log4j properties. Seems that does not require any code change.
>
> Rajat Khandelwal wrote:
> The way I see it, is that there can be an interface `QueryProgress` in lens-server-api and `LensDriver` can have a method returning `QueryProgress` The only thing `QueryProgress` *needs* to implement is `float/double getProgress()`. Each driver can have its own implementation, For example, doing a query status on hive query returns three rows, one of which is progress. The driver can parse and return that progress. Similar thing can be there for JDBC. CLI just needs to ask for `QueryProgress` object from server which in turn will ask it from the driver. Driver can give it's own implementation, which will eventually be returned by server to cli.
>
> Raju Bairishetti wrote:
> Yes, Right now Query status contains progress information as well. @Rajat Can we use the same for knowing the query progress information(i.e. QueryStatus.getProgessMessage()) or do we need to introduce a separate interface?
>
> Rajat Khandelwal wrote:
> Let's use what we already have :). But in cli's non verbose mode let's show the progress on cli in case of sync queries(http://stackoverflow.com/questions/852665/command-line-progress-bar-in-java). For async queries, `query progress` command can be there for the same purpose. `query status` prints the whole status.
>
> In verbose/debug mode, sync queries are showing the whole status, rest all remains the same.
>
> Amareshwari Sriramadasu wrote:
> Summarizing concerns and discussion above.
>
> We would like to have --verbose option to know more about whats happening a command, and --debug option to actually look at debug logs. Here is how it can be achiveed :
>
> - All client and CLI classes continues to use class level loggers for info, debug logging for debugging purpose. --debug option can add console appender to see the logs directly on console. If no option is available (as it is now), use can add console appendar in log4j, all the logs will start showing up now. But can be enhanced to add --debug option which sets the console appender.
> - For showing more information about a command, we need a separate logger than class level logger. --verbose option can add console appender for this separate verbose logger. Commands will be using this verbose logger to give intermediate information about the command.
>
> we are trying to add verbose option and verbose logger in this jira. If adding debug option has to be seperate jira, I'm fine with it.
>
> Does it make sense?
Thanks Amareshwari and Rajat for the responses. Will do the needful changes accordingly.
- Raju
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Himanshu Gahlaut <ga...@gmail.com>.
> On May 20, 2015, 5:23 a.m., Amareshwari Sriramadasu wrote:
> > I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
>
> Amareshwari Sriramadasu wrote:
> I meant while opening
> $lens-cli // can open in verbose loggind mode i.e. logging on console
> $lens-cli --quiet // can open in quiet logging mode i.e. no log on console.
>
> Rajat Khandelwal wrote:
> +1.
would completely agree to the fact that passing verbose flag to functions is a less optimal approach. Will try to post a suggestion soon.
- Himanshu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84467
-----------------------------------------------------------
I'm thinking, shall we make --verbose at lens-cli option, which initializes logger to console (if verbose is on) or to a file (otherwise). That would make logging available for all commands and we should not require passing verbose flag around and check for flag for logging allover.
- Amareshwari Sriramadasu
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On May 20, 2015, 4:50 a.m., Raju Bairishetti wrote:
> > lens-client/src/main/java/org/apache/lens/client/LensStatement.java, line 243
> > <https://reviews.apache.org/r/34457/diff/1/?file=964887#file964887line243>
> >
> > getting the driver name from driver className to know the driver in which query was submitted.
> >
> > Is there any better way to get the driver name?
I would say just print whatever is part of LensQuery. i.e. fully qualified driver name.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84462
-----------------------------------------------------------
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>
Re: Review Request 34457: Print the query execution status on CLI
Posted by Raju Bairishetti <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34457/#review84462
-----------------------------------------------------------
lens-client/src/main/java/org/apache/lens/client/LensStatement.java
<https://reviews.apache.org/r/34457/#comment135752>
getting the driver name from driver className to know the driver in which query was submitted.
Is there any better way to get the driver name?
- Raju Bairishetti
On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34457/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:48 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-25
> https://issues.apache.org/jira/browse/LENS-25
>
>
> Repository: lens
>
>
> Description
> -------
>
> Added a cli logger for showing the query execution status
> Added a verbose option
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 920ba9c
> lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java a48734a
> lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82
> tools/conf/client/log4j.properties 5520304
>
> Diff: https://reviews.apache.org/r/34457/diff/
>
>
> Testing
> -------
>
> Tested on cli
>
> **Enable logging by settign verbose value to true (by default to true)**
>
> **lens-shell>query execute cube select product_details.category, store_sales from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') --verbose true**
> 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver
> 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - RUNNING:Query is running in HiveServer! - Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}]
> 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - SUCCESSFUL:Query is successful! - Result Available
> product_details.category _c1
> Results of query stored at : file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows process in (27) seconds.
>
> **Disable logging by setting verbose value to false (by default to true)**
> lens-shell>query execute cube select sample_dim.name, sample_dim2.name from sample_dim where sample_dim.name != 'first' --verbose false
> sample_dim.name sample_dim2.name
> Results of query stored at : file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows process in (8) seconds.
>
>
> Thanks,
>
> Raju Bairishetti
>
>