You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Miklos Gergely <mg...@hortonworks.com> on 2019/05/08 10:24:39 UTC
Review Request 70608: Break up DDLTask - extract Partition related
operations
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70608/
-----------------------------------------------------------
Review request for hive and Zoltan Haindrich.
Bugs: HIVE-21703
https://issues.apache.org/jira/browse/HIVE-21703
Repository: hive-git
Description
-------
DDLTask is a huge class, more than 5000 lines long. The related DDLWork is also a huge class, which has a field for each DDL operation it supports. The goal is to refactor these in order to have everything cut into more handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl:
have a separate class for each operation
have a package for each operation group (database ddl, table ddl, etc), so the amount of classes under a package is more manageable
make all the requests (DDLDesc subclasses) immutable
DDLTask should be agnostic to the actual operations
right now let's ignore the issue of having some operations handled by DDLTask which are not actual DDL operations (lock, unlock, desc...)
In the interim time when there are two DDLTask and DDLWork classes in the code base the new ones in the new package are called DDLTask2 and DDLWork2 thus avoiding the usage of fully qualified class names where both the old and the new classes are in use.
Step #8: extract all the partition related operations from the old DDLTask, and move them under the new package.
Diffs
-----
hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzer.java 6b43b5333d
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java 66a1737839
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java 3820fabbf9
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AlterTableUtils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/AlterTableAddPartitionOperation.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/AlterTableAlterPartitionDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/AlterTableAlterPartitionOperation.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/AlterTableDropPartitionOperation.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/AlterTableExchangePartitionsOperation.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/AlterTableRenamePartitionDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/AlterTableRenamePartitionOperation.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/ShowPartitionsOperation.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/package-info.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 3d4ba0110a
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/PartitionEvent.java b59ab6c319
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/TableEvent.java 3dcc1d713c
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/FSPartitionEvent.java ee804e862a
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/FSTableEvent.java 27009f0385
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/ReplicationState.java 5a4dc4c5c0
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java 0add38b213
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 111cd1dc3f
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 3bf2a43b01
ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 38d9940bd0
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/DropPartitionHandler.java 5e88b6ebae
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/RenamePartitionHandler.java 9c66210e70
ql/src/java/org/apache/hadoop/hive/ql/plan/AddPartitionDesc.java 8ea857e956
ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableAlterPartDesc.java 652c007643
ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableExchangePartition.java 2a67494b00
ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 0505e07db4
ql/src/java/org/apache/hadoop/hive/ql/plan/DropPartitionDesc.java 81fcc4689d
ql/src/java/org/apache/hadoop/hive/ql/plan/RenamePartitionDesc.java b4edbfe633
ql/src/java/org/apache/hadoop/hive/ql/plan/ShowPartitionsDesc.java c9ed41a6e5
ql/src/test/queries/clientpositive/alter_partition_change_col.q c207731208
ql/src/test/queries/clientpositive/alter_rename_partition.q b6f6ccce3b
ql/src/test/queries/clientpositive/drop_partitions_filter.q 5862753b23
ql/src/test/queries/clientpositive/exchgpartition2lel.q 567ff8a0bc
ql/src/test/queries/clientpositive/show_partitions.q d22c483fea
ql/src/test/results/clientnegative/add_partition_with_whitelist.q.out 95da2195e9
ql/src/test/results/clientnegative/addpart1.q.out e1255bc0f9
ql/src/test/results/clientnegative/alter_partition_with_whitelist.q.out ae265eb6c9
ql/src/test/results/clientnegative/alter_rename_partition_failure.q.out 486ab8cdb0
ql/src/test/results/clientnegative/alter_rename_partition_failure2.q.out acb66acd43
ql/src/test/results/clientnegative/exchange_partition.q.out bfdf413d89
ql/src/test/results/clientnegative/external2.q.out 05ddc28820
ql/src/test/results/clientpositive/add_part_multiple.q.out 7631c4d02f
ql/src/test/results/clientpositive/alter_partition_change_col.q.out d330026392
ql/src/test/results/clientpositive/alter_rename_partition.q.out fc7d750b3b
ql/src/test/results/clientpositive/drop_deleted_partitions.q.out e2c4443055
ql/src/test/results/clientpositive/drop_multi_partitions.q.out 53978e8cc9
ql/src/test/results/clientpositive/drop_partitions_filter.q.out 2cbc05da5e
ql/src/test/results/clientpositive/llap/exchgpartition2lel.q.out 2ec3e1a4e7
ql/src/test/results/clientpositive/show_partitions.q.out 8b7473a182
ql/src/test/results/clientpositive/showparts.q.out ddb44b64e7
ql/src/test/results/clientpositive/spark/add_part_multiple.q.out 7631c4d02f
streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java fa7e079331
Diff: https://reviews.apache.org/r/70608/diff/1/
Testing
-------
Q tests are running, added some new tests too.
Thanks,
Miklos Gergely