[GitHub] [hadoop] adoroszlai opened a new pull request #1357: HDDS-2042. Avoid log on console with Ozone shell

[GitBox <gi...@apache.org>]

adoroszlai opened a new pull request #1357: HDDS-2042. Avoid log on console with Ozone shell
URL: https://github.com/apache/hadoop/pull/1357
 
 
   ## What changes were proposed in this pull request?
   
   Sample Docker Compose-based clusters are currently [configured](https://github.com/apache/hadoop/blob/3329257d99d2808e66ae6c2fe87a9c4f8877026f/hadoop-ozone/dist/src/main/compose/ozone/docker-config#L32-L33) to log almost everything to the console.  This is useful because logs can be accessed via `docker logs` or `docker-compose logs`.  Further, [default logs settings](https://github.com/apache/hadoop/blob/3329257d99d2808e66ae6c2fe87a9c4f8877026f/hadoop-ozone/dist/src/main/conf/log4j.properties#L137-L138) for non-Docker Compose clusters also direct INFO and higher level messages to the console.
   
   However, it does not work well with interactive clients, eg. `ozone sh`, since regular console output and log messages are mixed.  Previously there [were](https://issues.apache.org/jira/browse/HDDS-1489) [attempts](https://issues.apache.org/jira/browse/HDDS-465) to work around this by setting specific class log levels to `WARN` or `ERROR`.  This approach has two problems:
   1. it applies to Ozone server processes, too
   2. new INFO or higher level messages may be introduced any time (eg. MetricsSystem startup)
   
   This pull request proposes to collect logs for `ozone` subcommands `sh` and `freon` to files instead of the console.  It eliminates the need for per-class log level settings (though current ones are kept).
   
   https://issues.apache.org/jira/browse/HDDS-2042
   
   ## How was this patch tested?
   
   Tested commands in sample docker cluster.  Output is clean (except pesky warning (omitted here) about illegal access for key operations).
   
   ```
   $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozones3
   $ docker-compose exec scm bash
   bash-4.2$ ozone sh volume create vol1
   bash-4.2$ ozone sh bucket create vol1/buck1
   bash-4.2$ ozone sh key put vol1/buck1/key1 /etc/passwd
   ```
   
   Verified that log file contains the messages previously sent to console:
   
   ```
   bash-4.2$ tail /var/log/hadoop/ozone-shell.log
   2019-08-27 14:45:02,695 [main] INFO  RpcClient:293 - Creating Volume: vol1, with hadoop as owner.
   2019-08-27 14:45:13,099 [main] INFO  RpcClient:432 - Creating Bucket: vol1/buck1, with Versioning false and Storage Type set to DISK and Encryption set to false
   2019-08-27 14:45:24,011 [main] INFO  MetricsConfig:118 - Loaded properties from hadoop-metrics2.properties
   2019-08-27 14:45:24,160 [main] INFO  MetricsSystemImpl:374 - Scheduled Metric snapshot period at 10 second(s).
   2019-08-27 14:45:24,160 [main] INFO  MetricsSystemImpl:191 - XceiverClientMetrics metrics system started
   ```
   
   Verbose mode still works:
   
   ```
   bash-4.2$ ozone sh --verbose key put vol1/buck1/key2 /etc/passwd
   Volume Name : vol1
   Bucket Name : buck1
   Key Name : key2
   File Hash : b01f053617ddfeb782a3e757d9c08912
   ```
   
   Freon:
   
   ```
   bash-4.2$ ozone freon rk --numOfVolumes=1 --numOfBuckets=1 --numOfKeys=20 --numOfThreads=1
   
    100.00% |?????????????????????????????????????????????????????????????????????????????????????????????????????|  20/20 Time: 0:00:03
   
   ***************************************************
   Status: Success
   Git Base Revision: e97acb3bd8f3befd27418996fa5d4b50bf2e17bf
   Number of Volumes created: 1
   Number of Buckets created: 1
   Number of Keys added: 20
   Ratis replication factor: ONE
   Ratis replication type: STAND_ALONE
   Average Time spent in volume creation: 00:00:00,080
   Average Time spent in bucket creation: 00:00:00,013
   Average Time spent in key creation: 00:00:00,234
   Average Time spent in key write: 00:00:02,310
   Total bytes written: 204800
   Total Execution time: 00:00:06,284
   ***************************************************
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org