[jira] Commented: (HIVE-1359) Unit test should be shim-aware

["Ning Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870126#action_12870126 ] 

Ning Zhang commented on HIVE-1359:
----------------------------------

The proposal is to change the unit test directory structure to the following:

{code}
ql/src/test/queries/ clientpositive/common/  -- all .q files that should work on all Hadoop versions and whose outputs are the same
                                                      /shim/0.17 -- all .q files that only works on 0.17 and have different output than other Hadoop versions.
                                                                /0.18 -- ditto, could be a symlink to 0.17
                                                                /0.17 -- ditto coulbe a symlink to 0.17
                                                                / 0.20 -- .q files only works on hadoop 0.20 or output files are different from other hadoop versions.
ql/src/test/results/clientpositive/common/ -- .out files in the corresponding queries directory.
                                                    /shim/0.17 -- corresponding out file directories
{code}

The QTestUtil should be changed to be shim-aware: in addition to test common/*.q files, it also need to test .q files in the specific hadoop version. 

For committers, if there are multiple distinct set of test files (say 0.17 and 0.20), they need to test both versions before committing. 

{code}
  $ ant test -Dhadoop.version=0.17.2.1 ...
  $ ant test -Dhadoop.version=0.20.0 ...
{code}

For developers, if a new .q test is added and it is not shim-aware (i.e., does not depends on Hadoop version), it should be put into common only. If it is shim-aware (say having different behavior in Hadoop 0.20), it should be put in ALL shim subdirectories shim/0.17--0.20, but NOT common. The reason is that
common is tested for all hadoop versions and if it is put there different results will be expected. 


> Unit test should be shim-aware
> ------------------------------
>
>                 Key: HIVE-1359
>                 URL: https://issues.apache.org/jira/browse/HIVE-1359
>             Project: Hadoop Hive
>          Issue Type: New Feature
>            Reporter: Ning Zhang
>            Assignee: Ning Zhang
>
> Some features in Hive only works for certain Hadoop versions through shim. However the unit test structure is not shim-aware in that there is only one set of queries and expected outputs for all Hadoop versions. This may not be sufficient when we will have different output for different Hadoop versions. One example is CombineHiveInputFormat wich is only available from Hadoop 0.20. The plan using CombineHiveInputFormat and HiveInputFormat may be different. Another example is archival partitions (HAR) which is also only available from 0.20. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.