You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Anubhav Tarar <an...@knoldus.in> on 2017/07/13 13:03:56 UTC
Unable to Use Vectorized Reader In Hive
hi, i am uanble to use vectorized reader with hive 1.2.1 i use this link
from hive documentation
https://cwiki.apache.org/confluence/display/Hive/Vectorized+Query+Execution
step1: create table check(id int) stored as ORC;
OK
Time taken: 0.521 seconds
step2:hive> insert into check values(1);
Query ID = hduser_20170713162335_9bb368e6-c11f-4d87-8edf-86484fb0a233
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Job running in-process (local Hadoop)
2017-07-13 16:23:39,323 Stage-1 map = 0%, reduce = 0%
2017-07-13 16:23:40,390 Stage-1 map = 100%, reduce = 0%
Ended Job = job_local800495047_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to:
hdfs://localhost:54310/usr/hive/warehouse/check/.hive-staging_hive_2017-07-13_16-23-35_493_7294075038374724529-1/-ext-10000
Loading data to table default.check
Table default.check stats: [numFiles=1, numRows=1, totalSize=199,
rawDataSize=4]
MapReduce Jobs Launched:
Stage-Stage-1: HDFS Read: 4055 HDFS Write: 270 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
Time taken: 5.505 seconds
step3:
hive> explain select * from check where id >1;
OK
STAGE DEPENDENCIES:
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
TableScan
alias: check
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
stats: NONE
Filter Operator
predicate: (id > 1) (type: boolean)
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
Column stats: NONE
Select Operator
expressions: 1 (type: int)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
Column stats: NONE
explain command does not show that vectorized reader is applied
--
Thanks and Regards
* Anubhav Tarar *
* Software Consultant*
*Knoldus Software LLP <http://www.knoldus.com/home.knol> *
LinkedIn <http://in.linkedin.com/in/rahulforallp> Twitter
<https://twitter.com/RahulKu71223673> fb <ra...@facebook.com>
mob : 8588915184
Re: Unable to Use Vectorized Reader In Hive
Posted by Anubhav Tarar <an...@knoldus.in>.
so should it be update in hive cwiki that When Fetch is used in the plan
instead of Map, it do not vectorize.
On Thu, Jul 13, 2017 at 6:59 PM, Matthew McCline <mm...@hortonworks.com>
wrote:
>
> When Fetch is used in the plan instead of Map, we do not vectorize.
>
> Suggest using:
> set hive.fetch.task.conversion=none;
>
> ________________________________________
> From: Anubhav Tarar <an...@knoldus.in>
> Sent: Thursday, July 13, 2017 6:03 AM
> To: dev@hive.apache.org
> Subject: Unable to Use Vectorized Reader In Hive
>
> hi, i am uanble to use vectorized reader with hive 1.2.1 i use this link
> from hive documentation
>
> https://cwiki.apache.org/confluence/display/Hive/
> Vectorized+Query+Execution
>
> step1: create table check(id int) stored as ORC;
> OK
> Time taken: 0.521 seconds
>
> step2:hive> insert into check values(1);
> Query ID = hduser_20170713162335_9bb368e6-c11f-4d87-8edf-86484fb0a233
> Total jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Job running in-process (local Hadoop)
> 2017-07-13 16:23:39,323 Stage-1 map = 0%, reduce = 0%
> 2017-07-13 16:23:40,390 Stage-1 map = 100%, reduce = 0%
> Ended Job = job_local800495047_0001
> Stage-4 is selected by condition resolver.
> Stage-3 is filtered out by condition resolver.
> Stage-5 is filtered out by condition resolver.
> Moving data to:
> hdfs://localhost:54310/usr/hive/warehouse/check/.hive-
> staging_hive_2017-07-13_16-23-35_493_7294075038374724529-1/-ext-10000
> Loading data to table default.check
> Table default.check stats: [numFiles=1, numRows=1, totalSize=199,
> rawDataSize=4]
> MapReduce Jobs Launched:
> Stage-Stage-1: HDFS Read: 4055 HDFS Write: 270 SUCCESS
> Total MapReduce CPU Time Spent: 0 msec
> OK
> Time taken: 5.505 seconds
> step3:
> hive> explain select * from check where id >1;
> OK
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
>
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: check
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: (id > 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: 1 (type: int)
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
>
>
> explain command does not show that vectorized reader is applied
> --
> Thanks and Regards
>
> * Anubhav Tarar *
>
>
> * Software Consultant*
> *Knoldus Software LLP <http://www.knoldus.com/home.knol> *
> LinkedIn <http://in.linkedin.com/in/rahulforallp> Twitter
> <https://twitter.com/RahulKu71223673> fb <ra...@facebook.com>
> mob : 8588915184
>
>
>
--
Thanks and Regards
* Anubhav Tarar *
* Software Consultant*
*Knoldus Software LLP <http://www.knoldus.com/home.knol> *
LinkedIn <http://in.linkedin.com/in/rahulforallp> Twitter
<https://twitter.com/RahulKu71223673> fb <ra...@facebook.com>
mob : 8588915184
Re: Unable to Use Vectorized Reader In Hive
Posted by Matthew McCline <mm...@hortonworks.com>.
When Fetch is used in the plan instead of Map, we do not vectorize.
Suggest using:
set hive.fetch.task.conversion=none;
________________________________________
From: Anubhav Tarar <an...@knoldus.in>
Sent: Thursday, July 13, 2017 6:03 AM
To: dev@hive.apache.org
Subject: Unable to Use Vectorized Reader In Hive
hi, i am uanble to use vectorized reader with hive 1.2.1 i use this link
from hive documentation
https://cwiki.apache.org/confluence/display/Hive/Vectorized+Query+Execution
step1: create table check(id int) stored as ORC;
OK
Time taken: 0.521 seconds
step2:hive> insert into check values(1);
Query ID = hduser_20170713162335_9bb368e6-c11f-4d87-8edf-86484fb0a233
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Job running in-process (local Hadoop)
2017-07-13 16:23:39,323 Stage-1 map = 0%, reduce = 0%
2017-07-13 16:23:40,390 Stage-1 map = 100%, reduce = 0%
Ended Job = job_local800495047_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to:
hdfs://localhost:54310/usr/hive/warehouse/check/.hive-staging_hive_2017-07-13_16-23-35_493_7294075038374724529-1/-ext-10000
Loading data to table default.check
Table default.check stats: [numFiles=1, numRows=1, totalSize=199,
rawDataSize=4]
MapReduce Jobs Launched:
Stage-Stage-1: HDFS Read: 4055 HDFS Write: 270 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
Time taken: 5.505 seconds
step3:
hive> explain select * from check where id >1;
OK
STAGE DEPENDENCIES:
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
TableScan
alias: check
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
stats: NONE
Filter Operator
predicate: (id > 1) (type: boolean)
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
Column stats: NONE
Select Operator
expressions: 1 (type: int)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
Column stats: NONE
explain command does not show that vectorized reader is applied
--
Thanks and Regards
* Anubhav Tarar *
* Software Consultant*
*Knoldus Software LLP <http://www.knoldus.com/home.knol> *
LinkedIn <http://in.linkedin.com/in/rahulforallp> Twitter
<https://twitter.com/RahulKu71223673> fb <ra...@facebook.com>
mob : 8588915184