You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Slim Bouguerra <sb...@hortonworks.com> on 2018/10/25 16:27:14 UTC
Review Request 69170: HIVE-20486 Adding vectorized record reader for
Kafka input format
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69170/
-----------------------------------------------------------
Review request for hive, Gopal V and Teddy Choi.
Bugs: HIVE-20486
https://issues.apache.org/jira/browse/HIVE-20486
Repository: hive-git
Description
-------
This PR adds a vectorized record reader to the Kafka Storage Handler.
Minor documentation fixup
Some small refactors to the Serde index managements.
Diffs
-----
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 5924d06371
kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaInputFormat.java c401df9850
kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaRecordIterator.java 2225f19a4d
kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaRecordReader.java 7f8353c9f0
kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaSerDe.java 6b2ca1056e
kafka-handler/src/java/org/apache/hadoop/hive/kafka/SimpleKafkaWriter.java 678e190b3f
kafka-handler/src/java/org/apache/hadoop/hive/kafka/VectorizedKafkaRecordReader.java PRE-CREATION
kafka-handler/src/test/org/apache/hadoop/hive/kafka/SimpleKafkaWriterTest.java 8a9bbc7f66
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java e3e329f89b
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedSerde.java c97143c633
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java 7b788350b2
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcSerde.java f2295d6015
ql/src/test/queries/clientpositive/kafka_storage_handler.q e6cd276f95
ql/src/test/results/clientpositive/druid/kafka_storage_handler.q.out 8ea2aa9d3a
Diff: https://reviews.apache.org/r/69170/diff/1/
Testing
-------
Thanks,
Slim Bouguerra
Re: Review Request 69170: HIVE-20486 Adding vectorized record reader
for Kafka input format
Posted by Slim Bouguerra <sb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69170/
-----------------------------------------------------------
(Updated Oct. 25, 2018, 6:18 p.m.)
Review request for hive, Gopal V and Teddy Choi.
Changes
-------
remove extra refactoring
Bugs: HIVE-20486
https://issues.apache.org/jira/browse/HIVE-20486
Repository: hive-git
Description
-------
This PR adds a vectorized record reader to the Kafka Storage Handler.
Minor documentation fixup
Some small refactors to the Serde index managements.
Diffs (updated)
-----
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 5924d06371
kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaInputFormat.java c401df9850
kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaRecordIterator.java 2225f19a4d
kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaRecordReader.java 7f8353c9f0
kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaSerDe.java 6b2ca1056e
kafka-handler/src/java/org/apache/hadoop/hive/kafka/SimpleKafkaWriter.java 678e190b3f
kafka-handler/src/java/org/apache/hadoop/hive/kafka/VectorizedKafkaRecordReader.java PRE-CREATION
kafka-handler/src/test/org/apache/hadoop/hive/kafka/SimpleKafkaWriterTest.java 8a9bbc7f66
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java e3e329f89b
ql/src/test/queries/clientpositive/kafka_storage_handler.q e6cd276f95
ql/src/test/results/clientpositive/druid/kafka_storage_handler.q.out 8ea2aa9d3a
Diff: https://reviews.apache.org/r/69170/diff/3/
Changes: https://reviews.apache.org/r/69170/diff/2-3/
Testing
-------
Thanks,
Slim Bouguerra
Re: Review Request 69170: HIVE-20486 Adding vectorized record reader
for Kafka input format
Posted by Slim Bouguerra <sb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69170/
-----------------------------------------------------------
(Updated Oct. 25, 2018, 6:17 p.m.)
Review request for hive, Gopal V and Teddy Choi.
Changes
-------
removed the extra refactoring
Bugs: HIVE-20486
https://issues.apache.org/jira/browse/HIVE-20486
Repository: hive-git
Description
-------
This PR adds a vectorized record reader to the Kafka Storage Handler.
Minor documentation fixup
Some small refactors to the Serde index managements.
Diffs (updated)
-----
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/mr/HiveAccumuloRecordReader.java 45607cbecf
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/AccumuloPredicateHandler.java 0774d842ef
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/predicate/TestAccumuloPredicateHandler.java 0bb50e8784
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/predicate/TestAccumuloRangeGenerator.java 4975fa0d5e
common/src/java/org/apache/hadoop/hive/common/GcTimeMonitor.java edba6f9ad6
common/src/java/org/apache/hadoop/hive/common/JvmPauseMonitor.java 3c988da310
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java c3e7e5df8d
druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidRecordWriter.java 400262a107
druid-handler/src/java/org/apache/hadoop/hive/druid/json/KafkaSupervisorReport.java 5a6756ecbf
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidWritable.java 7390647c4b
druid-handler/src/test/org/apache/hadoop/hive/druid/TestDruidStorageHandler.java 510330d5d0
druid-handler/src/test/org/apache/hadoop/hive/ql/io/TestDruidRecordWriter.java cb8fa3919b
itests/hive-unit/src/test/java/org/apache/hadoop/hive/io/TestHadoopFileStatus.java 55877bea15
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersMoveWorkloadManager.java ad5aa180bf
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersTezSessionPoolManager.java faab11aa80
itests/qtest-druid/src/main/java/org/apache/hive/druid/ForkingDruidNode.java f81a0cae6b
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 5924d06371
kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaInputFormat.java c401df9850
kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaRecordIterator.java 2225f19a4d
kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaRecordReader.java 7f8353c9f0
kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaSerDe.java 6b2ca1056e
kafka-handler/src/java/org/apache/hadoop/hive/kafka/SimpleKafkaWriter.java 678e190b3f
kafka-handler/src/java/org/apache/hadoop/hive/kafka/VectorizedKafkaRecordReader.java PRE-CREATION
kafka-handler/src/test/org/apache/hadoop/hive/kafka/SimpleKafkaWriterTest.java 8a9bbc7f66
llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java 945474f540
llap-client/src/test/org/apache/hadoop/hive/llap/registry/impl/TestSlotZnode.java 0569505855
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java e3e329f89b
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 92775107bc
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 1a88b77fee
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java c55a3940c2
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java b14c8e4476
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java f8fa0cd1dd
ql/src/java/org/apache/hadoop/hive/ql/hooks/NoOperatorReuseCheckerHook.java 494459abd7
ql/src/java/org/apache/hadoop/hive/ql/hooks/PostExecOrcFileDump.java df99674f2c
ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java ed82d2d01e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 25b2d483d7
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/StreamUtils.java 215cefcd01
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java 4180dc471d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/RedundantDynamicPruningConditionsRemoval.java 4a60158892
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java 4d9963a061
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionTimeGranularityOptimizer.java 4297537adb
ql/src/java/org/apache/hadoop/hive/ql/optimizer/TablePropertyEnrichmentOptimizer.java a904182f91
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveSubQRemoveRelBuilder.java 05d1dc6cf2
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java fc9178f156
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java ed6659c6cc
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateReduceFunctionsRule.java 802c1c1fb2
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateReduceRule.java 30a918c9d3
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveDruidRules.java 7760cdc648
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java 600c7c0d07
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExpandDistinctAggregatesRule.java b33c4c51d9
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectRewriteRule.java 2499a83911
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java 9711625016
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinToMultiJoinRule.java ea5b06c0d4
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java 04800cca91
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java f7712e6c33
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java 238ae4ef4e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java b7c31bdfca
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSemiJoinRule.java 7799090d43
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java d1585c2901
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java 78b1281640
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 32fba6c8ff
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 82e975a50d
ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStats.java b723820f7b
ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsNoJobTask.java 53b3065a88
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java b7adc485a7
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java 2931c043e2
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java f046191ae4
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRowGroupFilter.java c33b701d94
ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/TestCBORuleFiredOnlyOnce.java e8dd57228f
ql/src/test/org/apache/hadoop/hive/ql/plan/TestExecutionEngineWorkConcurrency.java 1239ca5887
ql/src/test/queries/clientpositive/kafka_storage_handler.q e6cd276f95
ql/src/test/results/clientpositive/druid/kafka_storage_handler.q.out 8ea2aa9d3a
spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java 2be19de3f6
spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 8130860f2b
spark-client/src/test/java/org/apache/hive/spark/client/rpc/TestKryoMessageCodec.java fb736471b2
spark-client/src/test/java/org/apache/hive/spark/client/rpc/TestRpc.java 7da33fef2e
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 8cd46e3f44
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java 9c15804049
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONAddPartitionMessage.java 6494cb8dc7
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONCommitTxnMessage.java 482fc8e26b
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONCreateTableMessage.java 145ee4b199
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/JvmPauseMonitor.java 9a9e91b5e7
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/HiveSchemaHelper.java 2da07a5929
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 7429d18226
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java 4fc3688f2e
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetPartitions.java 4d7f7c1220
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetTableMeta.java 59daa526d8
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java a338bd4032
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/ExecutionController.java 29789c75ef
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/ExecutionPhase.java 7ab98f6a6c
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/HostExecutor.java 47347ebfd4
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/JIRAService.java 8eeb061c57
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/LogDirectoryCleaner.java a36b6f7c19
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 4e6aa6df96
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/Phase.java 34c66ce5db
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestParser.java ad6dad4ae8
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/UnitTestPropertiesParser.java a482fce8ed
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/AbstractTestPhase.java 71e1a64224
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/MockLocalCommandFactory.java 5de62bc13a
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/MockRSyncCommandExecutor.java fd4749eedf
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/MockSSHCommandExecutor.java 1f3db120ac
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.java 8964453d8b
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java 0ea51b7add
Diff: https://reviews.apache.org/r/69170/diff/2/
Changes: https://reviews.apache.org/r/69170/diff/1-2/
Testing
-------
Thanks,
Slim Bouguerra