You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Peter Vary <pv...@cloudera.com> on 2017/04/07 09:25:58 UTC
Re: Review Request 57693: HIVE-16146 If possible find a better way to
filter the TestBeeLineDriver outpu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57693/
-----------------------------------------------------------
(Updated April 7, 2017, 9:25 a.m.)
Review request for hive, Zoltan Haindrich, Marta Kuczora, Miklos Csanady, Naveen Gangam, Vihang Karajgaonkar, and Barna Zsombor Klara.
Changes
-------
After the review comments updated the patch with major changes:
- BeeLine - removed every previous changes, and added an inTestMode attribute accessbile only from the same package
- Commands - if the BeeLine client is in test mode print the operation logs regardless of other settings
- LogDivertAppenderForTest - For creating a second short log file, as Zoltan suggested
- OperationLog - to make it possible to return the short log file instead of the full one
- HiveConf - added 2 new configuration variables for tests:
-- hive.in.test.short.logs - to use the new short logs
-- hive.in.test.remove.logs - not to remove the logs after the session/query is finished
Some other smaller changes to make the above ones working.
I think this patch is ready to review after the big revamp
Bugs: HIVE-16146
https://issues.apache.org/jira/browse/HIVE-16146
Repository: hive-git
Description
-------
The goal was to generate '\0' markers around the raw log items we want to keep in the golden files.
To archive this I had to do a small functional change and some small refactor:
- Removed the immutability of the format map in BeeLine, so the test could add the QFile specific OutputFromat as a possible format
- The PostExecutePrinter and the PreExecutePrinter got a common ancestor, which was due anyway because PostExecutePrinter reused static methods from PreExecutePrinter. This way I was able to create QFile specific printers which are generating the desired markers.
- Moved the QFile test to the org.apache.hive.beeline package, so the test classes can use the package private classes and methods
For one reason or other BeeLine added an extra space character at the end of the lines for multiline commands - I have removed this space - Will see if this effects any unit test or not.
With the above mentioned *OutputFromat*, *QFilePreExecutePrinter*, *QFilePostExecutePrinter* we can mark the lines which are needed in the q.out file, and during the filtering we can remove the unneeded parts - I prefer to keep the log level high in the raw files, so in case of a test failure we can have better understanding of what has happened.
In the test files:
- Updated the beforeExecute, and afterExecute methods to set the new outputformat and the new hooks
- Removed the query specific filters, since they are non exitstent in the CLI tests
- Simplified the static filterset - currently only contains the filters which are really neccessary for the actual tests - might grow to the same size than in the QTestUtils - but if we do not want to run all of the test we would like to keep this list as small as possible
- Removed unnecessary configurations from QFileBuilder which will be not needed in case we want to mimic the CLI results
Diffs (updated)
-----
beeline/src/java/org/apache/hive/beeline/BeeLine.java 11526a7
beeline/src/java/org/apache/hive/beeline/Commands.java 2578728
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7d4a6a0
itests/src/test/resources/testconfiguration.properties 7a70c9c
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java 0d63f5d
itests/util/src/main/java/org/apache/hive/beeline/qfile/QFile.java ae5a349
itests/util/src/main/java/org/apache/hive/beeline/qfile/QFileBeeLineClient.java 760fde6
itests/util/src/main/java/org/apache/hive/beeline/qfile/package-info.java fcd50ec
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java a5c0fcd
ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppenderForTest.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java c37a633
ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 385f9b7
ql/src/test/results/clientpositive/beeline/escape_comments.q.out abc0fee
ql/src/test/results/clientpositive/beeline/select_dummy_source.q.out PRE-CREATION
service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f62ee4e
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 418f453
Diff: https://reviews.apache.org/r/57693/diff/2/
Changes: https://reviews.apache.org/r/57693/diff/1-2/
Testing
-------
Added a new simple query file from CLI driver, and checked that the generated output is the same
Thanks,
Peter Vary