You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Zoltan Haindrich <ki...@rxd.hu> on 2016/07/28 22:18:25 UTC
Review Request 50586: HIVE-14328 fix flapping qtests
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50586/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-14329
https://issues.apache.org/jira/browse/HIVE-14329
Repository: hive-git
Description
-------
main fix is some added sorting logic to MetaDataFormatUtils
i've seen some STATS_ACCURATE / COLUMN_STATS related order changes too; in that class:
* fix for StatsSetupConst#setBasicStatsState: failed to upgrade&set basic stats status
* removed: inconsitency: StatsSetupConst#setColumnStatsState would put lowercase colnames if record is new; original colnames during update/addition
i've choosed to be keep the column names **case sensitive**
* removed: upgrade-while-read from: StatsSetupConst#areBasicStatsUptoDate
i'm not sure but: ...would it make sense to use jackson/ObjectMapper to parse the params into a pojo; which then can be used more easily to read/write colstats and other metadata like it more easily - i haven't looked into it; how complicated would this be...
Diffs
-----
common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 01e6010
common/src/test/org/apache/hadoop/hive/common/TestStatsSetupConst.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java 03803bb
Diff: https://reviews.apache.org/r/50586/diff/
Testing
-------
waiting for 1st test run
Thanks,
Zoltan Haindrich
Re: Review Request 50586: HIVE-14329 fix flapping qtests
Posted by Zoltan Haindrich <ki...@rxd.hu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50586/
-----------------------------------------------------------
(Updated July 30, 2016, 8:22 p.m.)
Review request for hive.
Changes
-------
add new version of patch
Summary (updated)
-----------------
HIVE-14329 fix flapping qtests
Bugs: HIVE-14329
https://issues.apache.org/jira/browse/HIVE-14329
Repository: hive-git
Description (updated)
-------
fix hashmap order changes inside qtest outputs; most common is I think is skewed key order differences.
main fix is some added sorting logic to MetaDataFormatUtils
i've seen some STATS_ACCURATE / COLUMN_STATS related order changes too; in that class i've noticed some minor problems:
* fix for StatsSetupConst#setBasicStatsState: failed to upgrade&set basic stats status
* removed: inconsitency: StatsSetupConst#setColumnStatsState would put lowercase colnames if record is new; original colnames during update/addition
i've choosed to be keep the column names **case sensitive** - this is good?
* removed: upgrade-while-read from: StatsSetupConst#areBasicStatsUptoDate
even thru this may help avoiding STATS_ACCURATE related problems; it's not bulletproof - jsonobject tries hard to keep the insertion order inside its parts.
i'm not sure but: ...would it make sense to use jackson/ObjectMapper to parse the params into a pojo; which then can be used more easily to read/write colstats and other metadata like it more easily - i haven't looked into it; how complicated would this be...
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 01e6010
common/src/test/org/apache/hadoop/hive/common/TestStatsSetupConst.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java 03803bb
ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_1.q.out 00f3776
ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out df4a75b
ql/src/test/results/clientpositive/create_skewed_table1.q.out fe5ea0f
ql/src/test/results/clientpositive/extrapolate_part_stats_full.q.out 8f40040
ql/src/test/results/clientpositive/extrapolate_part_stats_partial.q.out 0acfe90
ql/src/test/results/clientpositive/extrapolate_part_stats_partial_ndv.q.out 70e4db3
ql/src/test/results/clientpositive/list_bucket_dml_10.q.out d4681b7
ql/src/test/results/clientpositive/list_bucket_dml_11.q.out ecf54a8
ql/src/test/results/clientpositive/list_bucket_dml_12.q.out 0e11f3f
ql/src/test/results/clientpositive/list_bucket_dml_13.q.out 93ebef0
ql/src/test/results/clientpositive/list_bucket_dml_2.q.out a29c224
ql/src/test/results/clientpositive/list_bucket_dml_4.q.out 5f0406a
ql/src/test/results/clientpositive/list_bucket_dml_5.q.out 09cb847
ql/src/test/results/clientpositive/list_bucket_dml_6.q.out e53fee7
ql/src/test/results/clientpositive/list_bucket_dml_8.q.out ee36d3f
ql/src/test/results/clientpositive/list_bucket_dml_9.q.out 81f3af3
ql/src/test/results/clientpositive/list_bucket_query_multiskew_1.q.out 492866a
ql/src/test/results/clientpositive/list_bucket_query_multiskew_2.q.out cfb613a
ql/src/test/results/clientpositive/list_bucket_query_multiskew_3.q.out 6a5f8f5
ql/src/test/results/clientpositive/list_bucket_query_oneskew_3.q.out 0c6fb63
ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.out 9eca85a
ql/src/test/results/clientpositive/stats_list_bucket.q.out c66da97
Diff: https://reviews.apache.org/r/50586/diff/
Testing (updated)
-------
r2: added qtest changes to patch
Thanks,
Zoltan Haindrich